Demo combiner

[Ostap]

Good news everyone

As everyone already know - on 42 - all recorded demos ...are split into parts
This is not without a reason ...
The game is old and buggy ... so when some players get bad time ...and will de-sync - demo will get damaged...
to avoid that - at some point were decided to limit the demo time ...(better have 1 damaged part ...than the whole demo)
At the moment - each recorded demo part - is 3 minutes long.

Well .. recently ...i've created a remedy to the fragmentation of the demos ...
From now on - every downloaded demo file with multiple parts - will have an application inside allowing you to combine the parts
into single demo file (i'm sure - who is recording a video - would appreciate it

The zipped file - is all the same - just 2 more items present:



directory /tools
with an application inside - 42DemoFileCombiner.exe
and a windows batch file to run it - combine_demos.bat

When you run the combine_demos.bat it will initiate the combining application ...and all demos inside unzipped directory will be combined into single file
note :
only 42amsterdam name formatted files will be used in process




Resulting file - will be the combination of all demos inside directory - good for longer playback.
If any of the demos are broken - you still get a half broken demo. It will not fix anything ...

My best advise - - check the demos manually - if they are all fine - then you can combine them
If not - remove the broken one ...AND all demos after it ... (well ..move them to other place atleast)
you can combine demos after broken part later

------------------------------------------------------
The program file included - is an open source one - you can download source code here - https://42amsterdam.net/samforum/dow...do=file&id=520
there are both original PHP and eventual C++ transcoded versions available
Would not do any good combining other than 42 demos - but as a part of other projects...
feel free to use it ...


There is a little but
Demos generated on 42 prior 30 Nov 2018 ...are a bit out of sync ...each part is missing few frames...
so when combining old demos - you'll get a few messages in the game log - about sync ...



Demos generated after 30 Nov - are perfectly sync .... and should give not errors whatsoever...
"correcton: ...demos generated after 2nd December 15:00 CET.... on SE there were still some missing frames"

[Marco]

Good news everyone

As everyone already know - on 42 - all recorded demos ...are split into parts
This is not without a reason ...
The game is old and buggy ... so when some players get bad time ...and will de-sync - demo will get damaged...
to avoid that - at some point were decided to limit the demo time ...(better have 1 damaged part ...than the whole demo)
At the moment - each recorded demo part - is 3 minutes long.

Well .. recently ...i've created a remedy to the fragmentation of the demos ...
From now on - every downloaded demo file with multiple parts - will have an application inside allowing you to combine the parts
into single demo file (i'm sure - who is recording a video - would appreciate it

The zipped file - is all the same - its just there are 2 more items present:



directory /tools
with an application inside - 42DemoFileCombiner.exe
and a windows batch file to run it - combine_demos.bat

When you run the combine_demos.bat it will initiate the combining application ...and all demos inside unzipped directory will be combined into single file
note :
only 42amsterdam name formatted files will be used in process




Resulting file - will be the combination of all demos inside directory - good for longer playback.
If any of the demos are broken - you still get a half broken demo. It will not fix anything ...

My best advise - - check the demos manually - if they are all fine - then you can combine them
If not - remove the broken one ...AND all demos after it ... (well ..move them to other place atleast)
you can combine demos after broken part later

------------------------------------------------------
The program file included - is an open source one - you can download source code here - https://42amsterdam.net/samforum/dow...do=file&id=520
there are both original PHP and eventual C++ transcoded versions available
Would not do any good combining other than 42 demos - but as a part of other projects...
feel free to use it ...


There is a little but
Demos generated on 42 prior 30 Nov 2018 ...are a bit out of sync ...each part is missing few frames...
so when combining old demos - you'll get a few messages in the game log - about sync ...



Demos generated after 30 Nov - are perfectly sync .... and should give not errors whatsoever...
"correcton: ...demos generated after 2nd December 15:00 CET.... on SE there were still some missing frames"

Hi Ostap, I have seen this thread a few months ago. There would be other threads I have to respond, but I don't have much time.


Ok, I think that as a Demo Combiner was released... Another tool to work in the opposite way is needed.

It would be a little OT, but there is an issue with demos from some maps. Some ones get de-synced and their variants not.

Little Trouble is indeed a BIG trouble when watching demos.

I have found 2007 demos of 1.05 that get de-synced after 1min 40secs on LT and there is no way to re-sync them!

Paradoxically, demos from Shotty Trouble played on 1.05 can be watched on a 1.07 copy of the game. [FALSE! See below]

I thought it was a dll issue, because there are some demos from another website I couldn't watch on Yodeller. After installing Fix42, they got re-synced!

This also happened to an old demo from 2015 (or 2016) when I played on PRO - Multimap* server with GrandPa from Germany and other casual people. I remember there were several corrupted parts, but I was watching them on another notebook. When I transferred that archive onto my netbook, last year, I was able to re-watch that demo. Same for another demo in Hole.


But this isn't the case of Little Trouble... I think there is an error on the map itself. Yes, it sometimes happens to Fortress (there is an old thread here with a de-synced demo published by Bloodnowsky, back in early 2017), but it is rare.

Then, my guess is that it couldn't be a dll issue for sure. Heh, after 6+ years I play SS multiplayer, I'm still noob and ignorant about this!
Looks like you have to draw another flow chart on how demo recordings work in Sam code!

So, a tool for splitting demos (that I assume it already exists) could help. But this tool must be customized, like: I want to split a demo in n parts of 1 min duration, or 30 secs duration, 2 mins, 5 mins, etc... So the de-synced parts could be cut out and then rebuild the entire demo with this Demo Combiner.

What do you think about this?


*That demo was recorded on Little Trouble as well.

Paradoxically, demos from Shotty Trouble played on 1.05 can be watched on a 1.07 copy of the game.

Ok, I wrote a fake news: the 1.05 demo of Shotty Trouble on 1.07 installation gets de-synced after 7 mins 15 secs. On the 1.05 installation it plays flawless, but LT demos are still de-synced.

[Ostap]

Demo Split ?

good idea - i'll make one later

[Ostap]

So, a tool for splitting demos (that I assume it already exists) could help. But this tool must be customized, like: I want to split a demo in n parts of 1 min duration, or 30 secs duration, 2 mins, 5 mins, etc... So the de-synced parts could be cut out and then rebuild the entire demo with this Demo Combiner.

What do you think about this?

Splitting will work as you described .... by 1 min ...30sec ...whatever ...
but you wont be able to glue them together after you'll identify the broken one ... - ANYTHING AFTER the broken part - need to be combined separately ....as separate demo ....it cannot be added to the previous part !

[Marco]

Splitting will work as you described .... by 1 min ...30sec ...whatever ...
but you wont be able to glue them together after you'll identify the broken one ... - ANYTHING AFTER the broken part - need to be combined separately ....as separate demo ....it cannot be added to the previous part !

So after splitting and combining, resulting in a creation of 2 different demos, will I get the popup "Missing frames" you showed in the first screenshot, then?


Sorry, me and my zone have been 18 days without telephone/internet line and I read this just a few days ago.

No problem for having different demo parts: I'd prefer 2-3 working demos with a duration of 7mins or 3min30secs per part rather than a corrupted one of 14 mins.

[Ostap]

So after splitting and combining, resulting in a creation of 2 different demos, will I get the popup "Missing frames" you showed in the first screenshot, then?


Sorry, me and my zone have been 18 days without telephone/internet line and I read this just a few days ago.

No problem for having different demo parts: I'd prefer 2-3 working demos with a duration of 7mins or 3min30secs per part rather than a corrupted one of 14 mins.

Inside demo there are recorded frames ....each one has time tick stamp. When just few ticks are missing - you'll get the "missing frame" message (even a single one)
When more than few are missing - demo will simply freeze ...until end.

As you said - after splitting to remove bad part - you'll get 2 demos ...... working ...no missing frames ...no warnings

[Marco]

Inside demo there are recorded frames ....each one has time tick stamp. When just few ticks are missing - you'll get the "missing frame" message (even a single one)
When more than few are missing - demo will simply freeze ...until end.

As you said - after splitting to remove bad part - you'll get 2 demos ...... working ...no missing frames ...no warnings

Aaah, then it is ok. Thanks.

[Ostap]

Actually - after all ...
You gave me an idea ....

Why not break recording of demo once bad sync is detected ? (and maybe mark it somehow ....that next demo is bad..)
then when all players have good sync - start recording again new demo ....

This way bad syncs will be isolated automatically

I think this idea has some sound ... i'll check into it

[Marco]

Actually - after all ...
You gave me an idea ....

Why not break recording of demo once bad sync is detected ? (and maybe mark it somehow ....that next demo is bad..)
then when all players have good sync - start recording again new demo ....

This way bad syncs will be isolated automatically

I think this idea has some sound ... i'll check into it

I think this is a realtime solution to apply server-side when demos automatically start recording.

If you want to make a try with an already recorded demo, try this one from last night.



Part 5 is corrupted after a few seconds. Basically the tool should locate when players re-sync again at only part 5. But still, you will need to cut at least after 10-15 seconds part 5 has started.

Or how is your tool supposed to locate desyncing and then resyncing inside a whole demo?


Hmm, I think we are now talking about 2 different things.


PS: I don't get why the screen says "Little Trouble", whether Jack and Lendru were playing on Fortress...

[Ostap]

I think this is a realtime solution to apply server-side when demos automatically start recording.

If you want to make a try with an already recorded demo, try this one from last night.



Part 5 is corrupted after a few seconds. Basically the tool should locate when players re-sync again at only part 5. But still, you will need to cut at least after 10-15 seconds part 5 has started.

Or how is your tool supposed to locate desyncing and then resyncing inside a whole demo?


Hmm, I think we are now talking about 2 different things.


PS: I don't get why the screen says "Little Trouble", whether Jack and Lendru were playing on Fortress...

We are talking about two different things
I meant - while demo is recording ...... server does know when de-sync occur
Not afterwards

[Ostap]

or ...
during demo recording simply drop into text file times when de-sync is present ...
when combining - this could be used to cut at those places

[Ostap]

Good news everyone

There is "Bad-sync" filter now implemented in demo combiner!

How it works:

During demo recording - server checks if there are any bad-syncs
when detected - it will record them into a CSV file (.bst file with same name as demo)



Recorded data:

• Client ID
• Sequence Index (time tick - there are 20 ticks in a second)
• Bad Sync Counter

as you can see - right at the begin - player with index 1 - had some struggle .... few bad syncs ...then good ...again few bad syncs ...good again .... 14 bad syncs ....eventually it counted up to 1200 ..and he got disconnected



When combining (click the "combine_demos_no_bad_sync.bat") - and when bad sync records are present - demo will be cut at that point

Result of combining - will be few playable files without any bad syncs...



Here are some examples of how bad sync may be distributed across demo parts and how demos are combined:

In case of "single damaged demo file":



In case of "few overlapping damaged demo files - variant 1"



In case of "few overlapping damaged demo files - variant 2"



Good part of this filtering - sequences until the first bad sync will be included in working demo
bad part - all sequences in current demo after bad sync section - cannot be re-combined - so if bad sync happens just 10 sec
after demo part starts - total loss is 2m 50 sec ...problem with re-combining in this case ...is that header of each source demo contains snapshot of the world ...when de-synced sequences are removed - anything behind - becomes invalid relative to the world condition when bad-sync started.
(to fix it a bit ....i'll check the possibility to lower recording time to 1-2 minutes to minimize the loss)


Note: file "combine_demos_no_bad_sync.bat" will be added to the zip archive only if there were bad syncs recorded ...(demos recorded before 21-apr-2019 10:00 - will have none)



Or maybe better solution would be "flush" demo recording after bad-sync are gone for more than 3 seconds for example...
This would allow to harvest good sequences that otherwise would be lost

[Ostap]

Or maybe better solution would be "flush" demo recording after bad-sync are gone for more than 3 seconds for example...
This would allow to harvest good sequences that otherwise would be lost

Done

Now demo part containing bad-syncs will be flushed (stopped and started again) ...after 2 seconds last bad sequence were recorded.
So .. theoretically - maximal lost time when only one player had bad sync - would be 62 seconds (assuming he will play till disconnected)

here is a first example of one:

[Marco]

Wow, good job Ostap!

I'm quite surprised that, with your last version of Demo Combiner, didn't find any bad sync in that demo I posted earlier at reply #9.

I mean, after I upgraded Fix42Mod to latest version, and then combined the demo, I didn't lose any part. The demo worked 100% with no desyncing.


I have another question now: if ".dem" files are archives where Serious Engine saves thousand frames in (while recording), how have you been able to "open" a demo archive and see that?

[Ostap]

I'm quite surprised that, with your last version of Demo Combiner, didn't find any bad sync in that demo I posted earlier at reply #9.
I mean, after I upgraded Fix42Mod to latest version, and then combined the demo, I didn't lose any part. The demo worked 100% with no desyncing.

It shouldn't .... that demo has no "bad-sync" times recorded .... frankly - i'm not sure how it got better

I have another question now: if ".dem" files are archives where Serious Engine saves thousand frames in (while recording), how have you been able to "open" a demo archive and see that?

Demo is not an archive ....its a binary file with bunch of data dumped into it ....(world state and all the player/monster actions that were present during the game)
If you know the structure of file - its not a problem to read it (structure of the file you can derive from C++ code in distributed 1.10 SDK of the game)