DOS ain't dead - Lame 3.99 DOS port

RayeR

CZ,
17.10.2011, 03:06

Lame 3.99 DOS port (Announce)

Hi,
I just built new LAME 3.99 mp3 encoder under DJGPP (minor changes in makefile needed, opt used: march=pentium)
http://rayer.ic.cz/download/download.htm#LAME

---
DOS gives me freedom to unlimited HW access.

RayeR

CZ,
17.10.2011, 19:43

@ RayeR

x264 0.118 DOS port

Post reply

Now I'm trying to compile x264 encoder but have some problems. I messed headers and makefile a lot to work with djgpp but now it's going to good end. Finally I have missing only one function:
char *strtok_r(char *s1, const char *s2, char **s3);
DJGPP string lib seems to be outdated so I need to supply some code instead of it. Any Idea before I will start searching and grabbing from some libc code?

C:\x264>make gcc -ox264.exe x264.o input/input.o input/timecode.o input/raw.o input/y4m.o out put/raw.o output/matroska.o output/matroske.o output/flv.o output/flv_bytestream .o filters/filters.o filters/video/video.o filters/video/source.o filters/video/ internal.o filters/video/resize.o filters/video/cache.o filters/video/fix_vfr_.o filters/video/select_e.o filters/video/crop.o filters/video/depth.o extras/geto pt.o libx264.a -lm libx264.a(common.o):common.c:(.text+0x1384): undefined reference to `_strtok_r' libx264.a(ratecont.o):ratecont.c:(.text+0x65b9): undefined reference to `_strtok _r' collect2: ld returned 1 exit status make.exe: *** [x264.exe] Error 1

---
DOS gives me freedom to unlimited HW access.

Rugxulo

Usono,
17.10.2011, 23:41

@ RayeR

x264 0.118 DOS port

Post reply

> Now I'm trying to compile x264 encoder but have some problems. I messed
> headers and makefile a lot to work with djgpp but now it's going to good
> end. Finally I have missing only one function:
> char *strtok_r(char *s1, const char *s2, char **s3);
> DJGPP string lib seems to be outdated so I need to supply some code instead
> of it. Any Idea before I will start searching and grabbing from some libc
> code?

strtok should be ANSI C, but "_r" suffix typically means "reentrant", i.e. multi-threading aware, possibly from POSIX. (But I'm far far far from a pro at C, POSIX, etc., so take it with a grain of salt.)

RayeR

CZ,
18.10.2011, 15:09

@ Rugxulo

x264 0.118 DOS port

Post reply

> strtok should be ANSI C, but "_r" suffix typically means "reentrant", i.e.
> multi-threading aware, possibly from POSIX. (But I'm far far far from a pro
> at C, POSIX, etc., so take it with a grain of salt.)

compilation done but... // forwarding from DJGPP google group:

> If you want less source level changes, you can define a macro,
> something like (untested):

> #define strtok_r(s1,s2,s3) strtok(s1,s2)

Thanks for explanation. I used the macro and it compiled OK. Maybe it
would be usefull to include this macro to djgpp string.h
Unfortunately when I tested compiled x264 to compress sample video it
crashes immediatelly:
C:\X264>X264.EXE -o out.flv example.y4m y4m [info]: 384x288p 0:0 @ 25/1 fps (cfr) x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2 x264 [info]: profile High, level 2.1 Exiting due to signal SIGSEGV General Protection Fault at eip=000d9e10 eax=00a39f10 ebx=00a1fdb0 ecx=000001c0 edx=00a39f10 esi=00226eb0 edi=00226eb0 ebp=00000000 esp=001f3eec program=C:\X264\X264.EXE cs: sel=01a7 base=029e0000 limit=00acffff ds: sel=01af base=029e0000 limit=00acffff es: sel=01af base=029e0000 limit=00acffff fs: sel=017f base=00005e10 limit=0000ffff gs: sel=01bf base=00000000 limit=0010ffff ss: sel=01af base=029e0000 limit=00acffff App stack: [001f4bb8..00174bbc] Exceptn stack: [00174adc..00172b9c]
(running under XP NTVDM)
maybe it's issue with sse or something else messed up...

I remember that here was some talking about SSE and crashing under DOS but I don't remember the result, any idea?

---
DOS gives me freedom to unlimited HW access.

RayeR

CZ,
18.10.2011, 15:38

@ RayeR

x264 0.118 DOS port

Post reply

> > C:\X264>X264.EXE -o out.flv example.y4m > y4m [info]: 384x288p 0:0 @ 25/1 fps (cfr) > x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle > SSE4.2 > x264 [info]: profile High, level 2.1 > Exiting due to signal SIGSEGV > General Protection Fault at eip=000d9e10 > eax=00a39f10 ebx=00a1fdb0 ecx=000001c0 edx=00a39f10 esi=00226eb0 > edi=00226eb0 > ebp=00000000 esp=001f3eec program=C:\X264\X264.EXE > cs: sel=01a7 base=029e0000 limit=00acffff > ds: sel=01af base=029e0000 limit=00acffff > es: sel=01af base=029e0000 limit=00acffff > fs: sel=017f base=00005e10 limit=0000ffff > gs: sel=01bf base=00000000 limit=0010ffff > ss: sel=01af base=029e0000 limit=00acffff > App stack: [001f4bb8..00174bbc] Exceptn stack: [00174adc..00172b9c] >

I had to disable HAVE_MMX in config.h that also turned off dynamic CPU detection and it's working now. It works regardless C opt. flags. There are also some ASM files compiled by YASM...

---
DOS gives me freedom to unlimited HW access.

Zyzzle

19.10.2011, 02:02

@ RayeR

x264 0.118 DOS port

Post reply

> I had to disable HAVE_MMX in config.h that also turned off dynamic CPU
> detection and it's working now. It works regardless C opt. flags. There are
> also some ASM files compiled by YASM...

I'd love to test the x264 encoder if you've got a working DOS port now. Any chance of posting the binary on your site?

BTW: I can't find any differences between your LAME 3.99 port and the old 3.98.3 DJGPP binary... Are there any speed improvements?

RayeR

CZ,
21.10.2011, 03:53

@ Zyzzle

x264 0.118 DOS port

Post reply

> I'd love to test the x264 encoder if you've got a working DOS port now. Any
> chance of posting the binary on your site?

You can get it here: http://rayer.ic.cz/download/x264.exe
But this test version is compiled without libavformat so input filter is very poor, you can compress only raw and mpeg .y4m video files. I'm fighting with libavformat but still have some missing functions during linking s it seems there are other depencies to libavutil and libavcodecand may be more...
There was also library hell caused by multiple placement of rdtsc code from djgpp time.h inline function so it doesn't go much smooth.

> BTW: I can't find any differences between your LAME 3.99 port and the old
> 3.98.3 DJGPP binary... Are there any speed improvements?

Don't expect significant speed inprovements. But some fixes was done, read changelog in oficial sources package.

---
DOS gives me freedom to unlimited HW access.

Khusraw

Bucharest, Romania,
21.10.2011, 13:53
(edited by Khusraw, 21.10.2011, 14:41)

@ RayeR

x264 0.118 DOS port

Post reply

> You can get it here: http://rayer.ic.cz/download/x264.exe
> But this test version is compiled without libavformat so input filter is
> very poor, you can compress only raw and mpeg .y4m video files. I'm
> fighting with libavformat but still have some missing functions during
> linking s it seems there are other depencies to libavutil and libavcodecand
> may be more...

What missing functions? It is possible that I still have some old libavformat patches which may fix the problem.

> There was also library hell caused by multiple placement of rdtsc code from
> djgpp time.h inline function so it doesn't go much smooth.

IIRC because it lacks the "static" keyword.

---
Glory to God for all things

RayeR

CZ,
25.10.2011, 09:54

@ Khusraw

x264 0.118 DOS port

Post reply

> What missing functions? It is possible that I still have some old
> libavformat patches which may fix the problem.

If you have preconfigured source package for DJGPP it would help...

> > djgpp time.h inline function so it doesn't go much smooth.
> > IIRC because it lacks the "static" keyword.

Hm, it should have, I'll suggest it in DJGPP group if someone will care and fix it in DJGPP tree.

---
DOS gives me freedom to unlimited HW access.

Khusraw

Bucharest, Romania,
25.10.2011, 13:04

@ RayeR

x264 0.118 DOS port

Post reply

> If you have preconfigured source package for DJGPP it would help...

I have one, but it is more than one year old. If you are interested I can send it to you.

---
Glory to God for all things

RayeR

CZ,
25.10.2011, 14:22

@ Khusraw

x264 0.118 DOS port

Post reply

> > If you have preconfigured source package for DJGPP it would help...
>
> I have one, but it is more than one year old. If you are interested I can
> send it to you.

I used libavcodec, libavformat, libavutil from ffmpeg-0.7.6 package, it's from summer 2011. I didn't give it up yet. I hope that this 3 libs are enough for x264, no more depencies...

---
DOS gives me freedom to unlimited HW access.

Khusraw

Bucharest, Romania,
25.10.2011, 15:42

@ RayeR

x264 0.118 DOS port

Post reply

> I used libavcodec, libavformat, libavutil from ffmpeg-0.7.6 package, it's
> from summer 2011. I didn't give it up yet. I hope that this 3 libs are
> enough for x264, no more depencies...

I think that the one I have is from ffmpeg-0.6 (libavformat 52.79.0), and I have never tried newer versions.

---
Glory to God for all things

Rugxulo

Usono,
18.10.2011, 19:25

@ RayeR

x264 0.118 DOS port

Post reply

> Unfortunately when I tested compiled x264 to compress sample video it
> crashes immediatelly:
> > C:\X264>X264.EXE -o out.flv example.y4m > y4m [info]: 384x288p 0:0 @ 25/1 fps (cfr) > x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle > SSE4.2 > x264 [info]: profile High, level 2.1 > Exiting due to signal SIGSEGV > General Protection Fault at eip=000d9e10 >

SIGSEGV isn't the same as SIGILL, so perhaps your cpu indeed supports SSE 4.2 (which isn't implemented that frequently on most average cpus). That would've been my first naive guess, but I can't say for sure.

> (running under XP NTVDM)
> maybe it's issue with sse or something else messed up...
>
> I remember that here was some talking about SSE and crashing under DOS but
> I don't remember the result, any idea?

You have to flip a bit in CR4 so that the OS can (ideally) know to save SSE state across task switches (FXSAVE, FXRSTOR). Otherwise it fails by default so that you won't depend on it being saved (when it's not). Obviously they expected all OSes these days to multitask, so it wasn't important to consider DOS, so DOS (despite being safe) couldn't use SSE until flipping that bit (while under ring 0 only).

Honestly, I would've almost rather had HIMEMX or JEMM386 or even KERNEL.SYS flip it for us, but they never did. Instead we've got HDPMI32 and CWSDPMI (r5 2008 or r7 2010) to do it. Then it works.

(And dare I say, SSE isn't exactly commonly used in DOS, and most compilers don't support generation for it. Even the ones that do are weak.)

> I had to disable HAVE_MMX in config.h that also turned off dynamic
> CPU detection and it's working now. It works regardless C opt.
> flags. There are also some ASM files compiled by YASM...

Yeah, hand-written assembly will almost always beat any (average) compiler at SIMD. Strange that HAVE_MMX disables dynamic detection of SSE as they are entirely different. It doesn't waste much time to check once via CPUID at startup, but admittedly, most people these days only use new-ish cpus, so they don't care (as Pentium4 and AMD64 both include SSE2, and those are both very common).

P.S. I don't think anybody ever bothered trying to support PAQ8 with SSE4's dot product instructions, though I always personally wondered if it would've helped or not (somewhat cynically doubt it, but who knows). I'm too dumb to figure it out myself, or at least I think so, so I haven't tried. I do actually have access to a SSE4 cpu these days, but I dunno, I'm too luddite / cynical to care almost, it's just frustrating to have ten bazillion technologies that are all arcane, fairly redundant, soon to be obsoleted, etc. (Intel is already working on AVX2. Ugh. Stop the madness!)

Khusraw

Bucharest, Romania,
21.10.2011, 14:56

@ RayeR

x264 0.118 DOS port

Post reply

> Unfortunately when I tested compiled x264 to compress sample video it
> crashes immediatelly:
> > C:\X264>X264.EXE -o out.flv example.y4m > y4m [info]: 384x288p 0:0 @ 25/1 fps (cfr) > x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle > SSE4.2 > x264 [info]: profile High, level 2.1 > Exiting due to signal SIGSEGV > General Protection Fault at eip=000d9e10 > eax=00a39f10 ebx=00a1fdb0 ecx=000001c0 edx=00a39f10 esi=00226eb0 > edi=00226eb0 > ebp=00000000 esp=001f3eec program=C:\X264\X264.EXE > cs: sel=01a7 base=029e0000 limit=00acffff > ds: sel=01af base=029e0000 limit=00acffff > es: sel=01af base=029e0000 limit=00acffff > fs: sel=017f base=00005e10 limit=0000ffff > gs: sel=01bf base=00000000 limit=0010ffff > ss: sel=01af base=029e0000 limit=00acffff > App stack: [001f4bb8..00174bbc] Exceptn stack: [00174adc..00172b9c] >
> (running under XP NTVDM)
> maybe it's issue with sse or something else messed up...
>
> I remember that here was some talking about SSE and crashing under DOS but
> I don't remember the result, any idea?

It is caused by the misalignment of SSE data from some of the assembler files, caused by DJGPP's COFF format limitations.

---
Glory to God for all things

RayeR

CZ,
25.10.2011, 09:58

@ Khusraw

x264 0.118 DOS port

Post reply

> It is caused by the misalignment of SSE data from some of the assembler
> files, caused by DJGPP's COFF format limitations.

Is it possible to enforce aligment with some __attribute__ or compiler option? Does it mean that when I call a SSE instruction on data that are not word or dword (I don't know the block size) aligned it will throw some exception and crashes?

---
DOS gives me freedom to unlimited HW access.

Khusraw

Bucharest, Romania,
25.10.2011, 13:11

@ RayeR

x264 0.118 DOS port

Post reply

> Is it possible to enforce aligment with some __attribute__ or compiler
> option? Does it mean that when I call a SSE instruction on data that are
> not word or dword (I don't know the block size) aligned it will throw some
> exception and crashes?

AFAIK it has to be aligned on 16 bytes (128 bits). I don't think that DJGPP's COFF supports such alignment, at least without some modifications.

---
Glory to God for all things

RayeR

CZ,
25.10.2011, 14:33

@ Khusraw

x264 0.118 DOS port

Post reply

> AFAIK it has to be aligned on 16 bytes (128 bits). I don't think that
> DJGPP's COFF supports such alignment, at least without some modifications.

I think that if I use SSE optimalization for GCC it would take care about it but the root of evil is in assebler files that are out of gcc control. If I disable HAVE_MMX in config but leave sse opts in makefile it works. It probably only not use assembler routines but still may use sse for other C-code. I didn't checked listing but binary is different size when use sse and when use mmx.

If I would want to use SSE in my code and need to have aligned memory I think it could be possible to use valloc() that should allocate page-aligned blocks of memory...

---
DOS gives me freedom to unlimited HW access.

Khusraw

Bucharest, Romania,
25.10.2011, 15:22
(edited by Khusraw, 25.10.2011, 15:48)

@ RayeR

x264 0.118 DOS port

Post reply

> I think that if I use SSE optimalization for GCC it would take care about
> it but the root of evil is in assebler files that are out of gcc control.
> If I disable HAVE_MMX in config but leave sse opts in makefile it works. It
> probably only not use assembler routines but still may use sse for other
> C-code. I didn't checked listing but binary is different size when use sse
> and when use mmx.

I was referring to the "untouched" assembler files, DJGPP's COFF sections are aligned to 16 bytes by default, so if you have the SSE data at the beginning of a section, it should work (for me it worked). IIRC HAVE_MMX failed because of the memalign problem I describe below. Personally I used #define memalign(a,b) memalign(b,a), and it finally worked.

> If I would want to use SSE in my code and need to have aligned memory I
> think it could be possible to use valloc() that should allocate
> page-aligned blocks of memory...

You can also use DJGPP's memalign, but it takes reversed parameters comparing to the one from Linux.

---
Glory to God for all things

RayeR

CZ,
25.10.2011, 16:31

@ Khusraw

x264 0.118 DOS port

Post reply

> You can also use DJGPP's memalign, but it takes reversed parameters
> comparing to the one from Linux.

Thanks, I didn't know this function, it maybe usefull later...

---
DOS gives me freedom to unlimited HW access.

Berlin, Germany,
17.10.2011, 22:38

@ RayeR

Lame 3.99 DOS port

Post reply

> Hi,
> I just built new LAME 3.99 mp3 encoder under DJGPP (minor changes in
> makefile needed, opt used: march=pentium)

What changes exactly? Mine to version 3.98.4 are documented in "lame-3.98.4.diff" (lam3984s.zip).

---
Forum admin

RayeR

CZ,
18.10.2011, 02:36

@ rr

Lame 3.99 DOS port

Post reply

> > Hi,
> > I just built new LAME 3.99 mp3 encoder under DJGPP (minor changes in
> > makefile needed, opt used: march=pentium)
>
> What changes exactly? Mine to version 3.98.4 are documented in
> "lame-3.98.4.diff"
> (lam3984s.zip).

I had problems with some gcc options like -mcpu -pipe -fmove-all-movables.
I replaced it with -march and removed 2 others...

---
DOS gives me freedom to unlimited HW access.

RayeR

CZ,
26.10.2011, 14:44

@ RayeR

New version of DJDEV 2.04

Post reply

BTW I solved problem with multiple code of _rdtsc from time.h - it's time to update DJDEV 2.04 to never version. According to post on DJGPP google group:

Ozkan Sezer
View profile
More options Oct 26, 9:56 am
Newsgroups: comp.os.msdos.djgpp
From: Ozkan Sezer <seze...@gmail.com>
Date: Wed, 26 Oct 2011 10:56:25 +0300
Local: Wed, Oct 26 2011 9:56 am
Subject: Re: _rdtsc(void) inline function in time.h - cause problem with multiple definition during linking
Reply | Reply to author | Forward | Print | Individual message | Show original | Report this message | Find messages by this author

- Hide quoted text -
- Show quoted text -
On Wed, Oct 26, 2011 at 5:48 AM, RayeR <gl...@centrum.cz> wrote:
>> BTW, is the issue with 2.03 or 2.04? IIRC, 2.04 had fix applied for this.

> I use djgpp 2.04 from official delorie site dir beta but time.h is
> pretty old: 16.2.2003, size: 2859B
> djdev204.zip 30-Nov-2003 15:01 2.1M

> Compiling options for x264 are:
> -Wshadow -O3 -ffast-math -Wall -I. -march=i686 -mfpmath=sse -msse -
> std=gnu99 -fomit-frame-pointer -fno-tree-vectorize -fno-zero-
> initialized-in-bss

> You mention that its problem of new gcc and inline and already that
> there's a fix.

> Is there available newer djdev binary package including fixes?
> Or what/where I should download some new fixed, but not too hot -
> quite stable, version to compile?

Try my own build if you want: http://uhexen2.sf.net/tmp/dj204/
Those I cross-built quickly on an x86_64-linux using gcc-3.4.6
and ran the mkdist script to generate the zips. djgpp stuff is
a CVS snapshot from 2011-10-02, stable for me, original source
tarball is there too.

---
DOS gives me freedom to unlimited HW access.