TOKENIZER.C
FILE INFORMATION
FILENAME(S): TOKENIZER.C
FILE TYPE(S): PRG
FILE SIZE: 1.9K
FIRST SEEN: 2025-11-30 21:07:08
APPEARS ON: 3 disk(s)
FILE HASH
461e68b0a5db958766dcac587b37ce64ecda639e84283e2aa11704c73f9d7548
FOUND ON DISKS (3 DISKS)
| DISK TITLE | FILENAME | FILE TYPE | COLLECTION | TRACK | SECTOR | ACTIONS |
|---|---|---|---|---|---|---|
| TPUG NOV 83 C | TOKENIZER.C | PRG | DuncanTwain | 3 | 19 | DOWNLOAD FILE |
| CCS64 1 2 | TOKENIZER.C | PRG | hedning, Mattias H00r | 12 | 9 | DOWNLOAD FILE |
| CCS 64 1 | TOKENIZER.C | PRG | hedning, Mattias H00r | 25 | 7 | DOWNLOAD FILE |
FILE CONTENT & ANALYSIS
00000000: 01 04 27 04 64 00 8F 20 54 4F 4B 45 4E 49 5A 45 |..'.d.. TOKENIZE| 00000010: 52 2C 20 44 41 56 49 44 20 57 49 4C 4C 49 41 4D |R, DAVID WILLIAM| 00000020: 53 2C 20 31 39 38 33 00 69 04 6E 00 8F 20 50 52 |S, 1983.i.n.. PR| 00000030: 4F 47 52 41 4D 20 54 4F 4B 45 4E 49 5A 45 53 20 |OGRAM TOKENIZES | 00000040: 4C 49 53 54 49 4E 47 53 20 4F 46 20 42 41 53 49 |LISTINGS OF BASI| 00000050: 43 20 50 52 4F 47 52 41 4D 53 20 28 56 45 52 59 |C PROGRAMS (VERY| 00000060: 20 53 4C 4F 57 4C 59 21 29 00 AA 04 73 00 8F 20 | SLOWLY!)...s.. | 00000070: 54 48 49 53 20 4D 41 59 20 42 45 20 55 53 45 46 |THIS MAY BE USEF| 00000080: 55 4C 20 49 4E 20 43 4F 4E 4E 45 43 54 49 4F 4E |UL IN CONNECTION| 00000090: 20 57 49 54 48 20 44 4F 57 4E 4C 4F 41 44 49 4E | WITH DOWNLOADIN| 000000A0: 47 20 50 52 4F 47 52 41 4D 53 00 C3 04 75 00 8F |G PROGRAMS...u..| 000000B0: 20 46 52 4F 4D 20 53 4F 4D 45 20 53 59 53 54 45 | FROM SOME SYSTE| 000000C0: 4D 53 2E 00 C9 04 78 00 3A 00 D6 04 82 00 86 20 |MS....x.:...... | 000000D0: 57 24 28 39 30 29 00 E5 04 8C 00 81 20 49 B2 30 |W$(90)...... I.0| 000000E0: 20 A4 20 39 30 00 F1 04 96 00 87 20 57 24 28 49 | . 90...... W$(I| 000000F0: 29 00 F7 04 A0 00 82 00 FD 04 AA 00 3A 00 1A 05 |)...........:...| 00000100: B4 00 85 20 22 4E 41 4D 45 20 4F 46 20 53 45 51 |... "NAME OF SEQ| 00000110: 20 46 49 4C 45 22 3B 53 46 24 00 2E 05 BE 00 99 | FILE";SF$......| 00000120: 20 22 44 52 49 56 45 20 23 20 3F 20 22 3B 00 52 | "DRIVE # ? ";.R| 00000130: 05 C8 00 A1 20 44 4E 24 3A 8B 20 44 4E 24 B3 22 |.... DN$:. DN$."| 00000140: 30 22 20 B0 20 44 4E 24 B1 22 31 22 20 A7 20 32 |0" . DN$."1" . 2| 00000150: 30 30 00 5C 05 D2 00 99 20 44 4E 24 00 6A 05 DC |00.\.... DN$.j..| 00000160: 00 9F 20 31 35 2C 38 2C 31 35 00 89 05 E6 00 9F |.. 15,8,15......| 00000170: 20 31 2C 38 2C 35 2C 44 4E 24 AA 22 3A 22 AA 53 | 1,8,5,DN$.":".S| 00000180: 46 24 AA 22 2C 53 2C 52 22 00 AC 05 F0 00 84 31 |F$.",S,R"......1| 00000190: 35 2C 45 2C 45 24 3A 8B 20 45 20 A7 20 99 20 45 |5,E,E$:. E . . E| 000001A0: 24 3A A0 20 31 35 3A 89 20 31 38 30 00 CD 05 FA |$:. 15:. 180....| 000001B0: 00 85 20 22 4E 41 4D 45 20 4F 46 20 50 52 4F 47 |.. "NAME OF PROG| 000001C0: 52 41 4D 20 46 49 4C 45 22 3B 50 46 24 00 E1 05 |RAM FILE";PF$...| 000001D0: 04 01 99 20 22 44 52 49 56 45 20 23 20 3F 20 22 |... "DRIVE # ? "| 000001E0: 3B 00 05 06 0E 01 A1 20 44 4E 24 3A 8B 20 44 4E |;...... DN$:. DN| 000001F0: 24 B3 22 30 22 20 B0 20 44 4E 24 B1 22 31 22 20 |$."0" . DN$."1" | 00000200: A7 20 32 37 30 00 0F 06 18 01 99 20 44 4E 24 00 |. 270...... DN$.| 00000210: 2E 06 22 01 9F 20 32 2C 38 2C 39 2C 44 4E 24 AA |..".. 2,8,9,DN$.| 00000220: 22 3A 22 AA 50 46 24 AA 22 2C 50 2C 57 22 00 50 |":".PF$.",P,W".P| 00000230: 06 2C 01 84 31 35 2C 45 2C 45 24 3A 8B 20 45 20 |.,..15,E,E$:. E | 00000240: A7 20 99 20 45 24 3A A0 20 32 3A 89 20 32 35 30 |. . E$:. 2:. 250| 00000250: 00 72 06 36 01 98 32 2C C7 28 31 29 3B C7 28 34 |.r.6..2,.(1);.(4| 00000260: 29 3B 3A 8F 20 53 54 41 52 54 20 41 44 44 52 45 |);:. START ADDRE| 00000270: 53 53 00 78 06 40 01 3A 00 8E 06 4A 01 8F 20 53 |SS.x.@.:...J.. S| 00000280: 54 41 52 54 20 4F 46 20 41 20 4C 49 4E 45 00 98 |TART OF A LINE..| 00000290: 06 54 01 4C 24 B2 22 22 00 A7 06 5E 01 8B 20 53 |.T.L$.""...^.. S| 000002A0: 53 20 A7 20 37 31 30 00 B8 06 68 01 A1 23 31 2C |S . 710...h..#1,| 000002B0: 47 24 3A 53 53 B2 53 54 00 D9 06 72 01 8B 20 47 |G$:SS.ST...r.. G| 000002C0: 24 B3 B1 C7 28 31 33 29 20 A7 20 4C 24 B2 4C 24 |$...(13) . L$.L$| 000002D0: AA 47 24 3A 89 20 33 35 30 00 E2 06 7C 01 99 20 |.G$:. 350...|.. | 000002E0: 4C 24 00 07 07 86 01 8B 20 C8 28 4C 24 2C 31 29 |L$...... .(L$,1)| 000002F0: B2 22 20 22 20 A7 20 4C 24 B2 CA 28 4C 24 2C 32 |." " . L$..(L$,2| 00000300: 29 3A 89 20 33 39 30 00 19 07 90 01 8B 20 4C 24 |):. 390...... L$| 00000310: B2 22 22 20 A7 20 33 33 30 00 2B 07 9A 01 8F 20 |."" . 330.+.... | 00000320: 4C 49 4E 45 20 4E 55 4D 42 45 52 00 37 07 A4 01 |LINE NUMBER.7...| 00000330: 41 B2 C6 28 4C 24 29 00 4F 07 AE 01 8B 20 41 B3 |A..(L$).O.... A.| 00000340: 34 38 20 B0 20 41 B1 35 37 20 A7 20 36 39 30 00 |48 . A.57 . 690.| 00000350: 5D 07 B8 01 4E 24 B2 22 22 3A 49 B2 31 00 6E 07 |]...N$."":I.1.n.| 00000360: C2 01 49 24 B2 CA 28 4C 24 2C 49 2C 31 29 00 9D |..I$..(L$,I,1)..| 00000370: 07 CC 01 8B 20 49 24 B1 B2 22 20 22 20 AF 20 49 |.... I$.." " . I| 00000380: 24 B3 B2 22 39 22 20 A7 20 4E 24 B2 4E 24 AA 49 |$.."9" . N$.N$.I| 00000390: 24 3A 49 B2 49 AA 31 3A 89 20 34 35 30 00 C3 07 |$:I.I.1:. 450...| 000003A0: D6 01 98 32 2C C7 28 35 30 29 3B C7 28 35 30 29 |...2,.(50);.(50)| 000003B0: 3B 3A 8F 20 44 55 4D 4D 59 20 4C 49 4E 45 20 4C |;:. DUMMY LINE L| 000003C0: 49 4E 4B 00 E4 07 E0 01 4E B2 C5 28 4E 24 29 3A |INK.....N..(N$):| 000003D0: 48 25 B2 4E AD 32 35 36 3A 4C 25 B2 4E AB 32 35 |H%.N.256:L%.N.25| 000003E0: 36 AC 48 25 00 F8 07 EA 01 98 32 2C C7 28 4C 25 |6.H%......2,.(L%| 000003F0: 29 3B C7 28 48 25 29 3B 00 18 08 F4 01 4C 24 B2 |);.(H%);.....L$.| 00000400: CA 28 4C 24 2C 49 29 3A 8F 20 43 55 54 20 4F 46 |.(L$,I):. CUT OF| 00000410: 46 20 4C 49 4E 45 20 23 00 1E 08 FE 01 3A 00 32 |F LINE #.....:.2| 00000420: 08 08 02 8F 20 54 4F 4B 45 4E 49 5A 45 20 54 45 |.... TOKENIZE TE| 00000430: 58 54 00 3B 08 12 02 51 46 B2 30 00 44 08 1C 02 |XT.;...QF.0.D...| 00000440: 50 54 B2 31 00 57 08 26 02 41 53 B2 C6 28 CA 28 |PT.1.W.&.AS..(.(| 00000450: 4C 24 2C 50 54 29 29 00 6D 08 30 02 8B 20 41 53 |L$,PT)).m.0.. AS| 00000460: B2 33 34 20 A7 20 51 46 B2 A8 20 51 46 00 89 08 |.34 . QF.. QF...| 00000470: 3A 02 8B 20 41 53 B2 35 38 20 A7 20 8B 20 51 46 |:.. AS.58 . . QF| 00000480: B2 33 20 A7 20 51 46 B2 30 00 98 08 44 02 8B 20 |.3 . QF.0...D.. | 00000490: 51 46 20 A7 20 36 35 30 00 A8 08 4E 02 81 20 57 |QF . 650...N.. W| 000004A0: 4E B2 30 20 A4 20 39 30 00 CF 08 58 02 8B 20 CA |N.0 . 90...X.. .| 000004B0: 28 4C 24 2C 50 54 2C C3 28 57 24 28 57 4E 29 29 |(L$,PT,.(W$(WN))| 000004C0: 29 B3 B1 57 24 28 57 4E 29 20 A7 20 36 34 30 00 |)..W$(WN) . 640.| 000004D0: 00 09 62 02 4C 24 B2 C8 28 4C 24 2C 50 54 AB 31 |..b.L$..(L$,PT.1| 000004E0: 29 AA C7 28 31 32 38 20 B0 20 57 4E 29 AA CA 28 |)..(128 . WN)..(| 000004F0: 4C 24 2C 50 54 AA C3 28 57 24 28 57 4E 29 29 29 |L$,PT..(W$(WN)))| 00000500: 00 35 09 6C 02 8B 20 57 4E B2 31 35 20 B0 20 57 |.5.l.. WN.15 . W| 00000510: 4E B2 33 20 A7 20 51 46 B2 57 4E 3A 8F 20 27 52 |N.3 . QF.WN:. 'R| 00000520: 45 4D 27 20 4F 52 20 27 44 41 54 41 27 20 49 4E |EM' OR 'DATA' IN| 00000530: 20 4C 49 4E 45 00 3F 09 76 02 57 4E B2 39 30 00 | LINE.?.v.WN.90.| 00000540: 48 09 80 02 82 20 57 4E 00 65 09 8A 02 50 54 B2 |H.... WN.e...PT.| 00000550: 50 54 AA 31 3A 8B 50 54 B3 B2 C3 28 4C 24 29 20 |PT.1:.PT...(L$) | 00000560: A7 20 35 35 30 00 70 09 94 02 98 32 2C 4C 24 3B |. 550.p....2,L$;| 00000570: 00 76 09 9E 02 3A 00 83 09 A8 02 98 32 2C C7 28 |.v...:......2,.(| 00000580: 30 29 3B 00 94 09 B2 02 8B 20 53 53 B2 30 20 A7 |0);...... SS.0 .| 00000590: 20 33 33 30 00 9A 09 BC 02 3A 00 AC 09 C6 02 98 | 330.....:......| 000005A0: 32 2C C7 28 30 29 3B C7 28 30 29 3B 00 BA 09 D0 |2,.(0);.(0);....| 000005B0: 02 A0 31 3A A0 32 3A A0 31 35 00 C0 09 DA 02 80 |..1:.2:.15......| 000005C0: 00 C6 09 E4 02 3A 00 FD 09 EE 02 83 20 45 4E 44 |.....:...... END| 000005D0: 2C 46 4F 52 2C 4E 45 58 54 2C 44 41 54 41 2C 49 |,FOR,NEXT,DATA,I| 000005E0: 4E 50 55 54 23 2C 49 4E 50 55 54 2C 44 49 4D 2C |NPUT#,INPUT,DIM,| 000005F0: 52 45 41 44 2C 4C 45 54 2C 47 4F 54 4F 00 35 0A |READ,LET,GOTO.5.| 00000600: F8 02 83 20 52 55 4E 2C 49 46 2C 52 45 53 54 4F |... RUN,IF,RESTO| 00000610: 52 45 2C 47 4F 53 55 42 2C 52 45 54 55 52 4E 2C |RE,GOSUB,RETURN,| 00000620: 52 45 4D 2C 53 54 4F 50 2C 4F 4E 2C 57 41 49 54 |REM,STOP,ON,WAIT| 00000630: 2C 4C 4F 41 44 00 6F 0A 02 03 83 20 53 41 56 45 |,LOAD.o.... SAVE| 00000640: 2C 56 45 52 49 46 59 2C 44 45 46 2C 50 4F 4B 45 |,VERIFY,DEF,POKE| 00000650: 2C 50 52 49 4E 54 23 2C 50 52 49 4E 54 2C 43 4F |,PRINT#,PRINT,CO| 00000660: 4E 54 2C 4C 49 53 54 2C 43 4C 52 2C 43 4D 44 00 |NT,LIST,CLR,CMD.| 00000670: A1 0A 0C 03 83 20 53 59 53 2C 4F 50 45 4E 2C 43 |..... SYS,OPEN,C| 00000680: 4C 4F 53 45 2C 47 45 54 2C 4E 45 57 2C 54 41 42 |LOSE,GET,NEW,TAB| 00000690: 28 2C 54 4F 2C 46 4E 2C 53 50 43 28 2C 54 48 45 |(,TO,FN,SPC(,THE| 000006A0: 4E 00 C3 0A 16 03 83 20 4E 4F 54 2C 53 54 45 50 |N...... NOT,STEP| 000006B0: 2C 2B 2C 2D 2C 2A 2C 2F 2C 5E 2C 41 4E 44 2C 4F |,+,-,*,/,^,AND,O| 000006C0: 52 2C 3E 00 ED 0A 20 03 83 20 3D 2C 3C 2C 53 47 |R,>... .. =,<,SG| 000006D0: 4E 2C 49 4E 54 2C 41 42 53 2C 55 53 52 2C 46 52 |N,INT,ABS,USR,FR| 000006E0: 45 2C 50 4F 53 2C 53 51 52 2C 52 4E 44 00 1D 0B |E,POS,SQR,RND...| 000006F0: 2A 03 83 20 4C 4F 47 2C 45 58 50 2C 43 4F 53 2C |*.. LOG,EXP,COS,| 00000700: 53 49 4E 2C 54 41 4E 2C 41 54 4E 2C 50 45 45 4B |SIN,TAN,ATN,PEEK| 00000710: 2C 4C 45 4E 2C 53 54 52 24 2C 56 41 4C 00 5C 0B |,LEN,STR$,VAL.\.| 00000720: 34 03 83 20 41 53 43 2C 43 48 52 24 2C 4C 45 46 |4.. ASC,CHR$,LEF| 00000730: 54 24 2C 52 49 47 48 54 24 2C 4D 49 44 24 2C 47 |T$,RIGHT$,MID$,G| 00000740: 4F 2C 43 4F 4E 43 41 54 2C 44 4F 50 45 4E 2C 44 |O,CONCAT,DOPEN,D| 00000750: 43 4C 4F 53 45 2C 52 45 43 4F 52 44 00 A7 0B 3E |CLOSE,RECORD...>| 00000760: 03 83 20 48 45 41 44 45 52 2C 43 4F 4C 4C 45 43 |.. HEADER,COLLEC| 00000770: 54 2C 42 41 43 4B 55 50 2C 43 4F 50 59 2C 41 50 |T,BACKUP,COPY,AP| 00000780: 50 45 4E 44 2C 44 53 41 56 45 2C 44 4C 4F 41 44 |PEND,DSAVE,DLOAD| 00000790: 2C 43 41 54 41 4C 4F 47 2C 52 45 4E 41 4D 45 2C |,CATALOG,RENAME,| 000007A0: 53 43 52 41 54 43 48 00 B7 0B 48 03 83 20 44 49 |SCRATCH...H.. DI| 000007B0: 52 45 43 54 4F 52 59 00 00 00 |RECTORY... |
..'.D.. TOKENIZER, DAVID WILLIAMS, 1983. I.N.. PROGRAM TOKENIZES LISTINGS OF BASI C PROGRAMS (VERY SLOWLY!)...S.. THIS MAY BE USEFUL IN CONNECTION WITH DOWNLOADIN G PROGRAMS.C.U.. FROM SOME SYSTEMS..I.X. :.V.... W$(90)...... I.0 . 90...... W$(I )... .......:...... "NAME OF SEQ FILE";S F$...... "DRIVE # ? ";.R.H.. DN$:. DN$." 0" . DN$."1" . 200.\.R.. DN$.J.... 15,8, 15...... 1,8,5,DN$.":".SF$.",S,R"......1 5,E,E$:. E . . E$: 15:. 180.M.... "NAME OF PROGRAM FILE";PF$...... "DRIVE # ? " ;...... DN$:. DN$."0" . DN$."1" . 270... ... DN$...".. 2,8,9,DN$.":".PF$.",P,W".P .,..15,E,E$:. E . . E$: 2:. 250.R.6..2, G(1);G(4);:. START ADDRESS.X.@.:...J.. S TART OF A LINE...T.L$.""...^.. SS . 710. ..H..#1,G$:SS.ST.Y.R.. G$..G(13) . L$.L$ .G$:. 350...|.. L$...... H(L$,1)." " . L $.J(L$,2):. 390...... L$."" . 330.+.... LINE NUMBER.7...A.F(L$).O.... A.48 . A.5 7 . 690.]...N$."":I.1.N.B.I$.J(L$,I,1).. .L.. I$.." " . I$.."9" . N$.N$.I$:I.I.1: . 450.C.V..2,G(50);G(50);:. DUMMY LINE L INK.....N.E(N$):H%.N.256:L%.N.256.H%.... ..2,G(L%);G(H%);.....L$.J(L$,I):. CUT OF F LINE #.....:.2.... TOKENIZE TEXT.;...Q F.0.D...PT.1.W.&.AS.F(J(L$,PT)).M.0.. AS .34 . QF.. QF...:.. AS.58 . . QF.3 . QF. 0...D.. QF . 650...N.. WN.0 . 90.O.X.. J (L$,PT,C(W$(WN)))..W$(WN) . 640...B.L$.H (L$,PT.1).G(128 . WN).J(L$,PT.C(W$(WN))) .5.L.. WN.15 . WN.3 . QF.WN:. 'REM' OR ' DATA' IN LINE.?.V.WN.90.H.... WN.E...PT. PT.1:.PT..C(L$) . 550.P....2,L$;.V...:.. ....2,G(0);...... SS.0 . 330.....:...F.. 2,G(0);G(0);...P. 1: 2: 15...Z...F...:.. .... END,FOR,NEXT,DATA,INPUT#,INPUT,DIM, READ,LET,GOTO.5.... RUN,IF,RESTORE,GOSUB ,RETURN,REM,STOP,ON,WAIT,LOAD.O.... SAVE ,VERIFY,DEF,POKE,PRINT#,PRINT,CONT,LIST, CLR,CMD...... SYS,OPEN,CLOSE,GET,NEW,TAB (,TO,FN,SPC(,THEN.C.... NOT,STEP,+,-,*,/ ,^,AND,OR,>... .. =,<,SGN,INT,ABS,USR,FR E,POS,SQR,RND...*.. LOG,EXP,COS,SIN,TAN, ATN,PEEK,LEN,STR$,VAL.\.4.. ASC,CHR$,LEF T$,RIGHT$,MID$,GO,CONCAT,DOPEN,DCLOSE,RE CORD...>.. HEADER,COLLECT,BACKUP,COPY,AP PEND,DSAVE,DLOAD,CATALOG,RENAME,SCRATCH. ..H.. DIRECTORY...
> CLICK IMAGE PREVIEW FOR FULL MODAL