![[BACK]](/cvsweb/icons/back.gif){kind=icon}
Return to keepnews CVS log ![[TXT]](/cvsweb/icons/text.gif){kind=icon}
![[DIR]](/cvsweb/icons/dir.gif){kind=icon}
Up to [CSRG BSD Unix] / 43BSD / contrib / news / misc|
Return to keepnews CVS log | Up to [CSRG BSD Unix] / 43BSD / contrib / news / misc |
1.1 ! root 1: From [email protected] (Chuq Von Rospach) Thu Jun 6 20:36:39 1985 ! 2: Relay-Version: version B 2.10.3 4.3bsd-beta 6/6/85; site seismo.UUCP ! 3: Posting-Version: version B 2.10.2 9/17/84 chuqui version 1.7 9/23/84; site nsc.UUCP ! 4: Path: seismo!nsc!chuqui ! 5: From: [email protected] (Chuq Von Rospach) ! 6: Newsgroups: net.sources ! 7: Subject: YA News Archiver ! 8: Message-ID: <[email protected]> ! 9: Date: 7 Jun 85 00:36:39 GMT ! 10: Date-Received: 7 Jun 85 06:25:58 GMT ! 11: Distribution: net ! 12: Organization: The Blue Parrot ! 13: Lines: 566 ! 14: ! 15: Here is a netnews archiver similar to the recently posted keepnews but ! 16: designed to work with much larger archives where the wonderful quadratic ! 17: search time feature of the Unix (Unix is a trademark of AT&T Bell Labs, ! 18: quadratic search times are a feature of Unix) becomes a real problem. This ! 19: archive also knows how to walk through a directory tree so you can simply ! 20: set it on /usr/spool/oldnews and let it do its work. There are lots of ! 21: other nifty things I call features (and you might, too) that make it a lot ! 22: easier to use than anything else I've seen set up to work on archives. Mine ! 23: simply outgrew any capability to do anything with about the same time I got ! 24: a request for information out of it. I found out (the hard way) that ! 25: keepnews wasn't terribly reliable working under 2.10.2, so I finally ! 26: decided to hack together my own. ! 27: ! 28: Comments, enhancements, bug fixes, etc... are welcome, but I can only work ! 29: on them on a time available basis... ! 30: ! 31: chuq ! 32: ------- ! 33: # This is a shell archive. ! 34: # Remove everything above and including the cut line. ! 35: # Then run the rest of the file through sh. ! 36: #-----cut here-----cut here-----cut here-----cut here----- ! 37: #!/bin/sh ! 38: # shar: Shell Archiver ! 39: # Run the following text with /bin/sh to create: ! 40: # README ! 41: # Makefile ! 42: # savenews.c ! 43: # This archive created: Thu Jun 6 17:28:50 1985 ! 44: # By: Chuq Von Rospach (The Blue Parrot) ! 45: cat << \SHAR_EOF > README ! 46: Savenews -- ! 47: ! 48: Savenews is a short program designed to make handling of usenet archives ! 49: generated by 'expire -a' easier, and to make it possible to find stuff in ! 50: the archive once it is there. ! 51: ! 52: It was created by me when I had to get something out of my archives and ! 53: realized that there was no way I was going to find anything in 70 megabytes ! 54: of random data. It keeps a set of logs of the Subject lines of the articles ! 55: and stores the articles themselves in a hashed subdirectory format designed ! 56: to minimize the quadratic lookup hassles of the unix directory system ! 57: (This, of course, is a feature). ! 58: ! 59: It has been put into the public domain by national semiconductor, and ! 60: neither myself or national guarantee that this code even exists, much ! 61: less that it does anything useful. This, BTW, is a disclaimer. ! 62: ! 63: chuq von rospach ! 64: national semiconductor ! 65: nsc!chuqui ! 66: SHAR_EOF ! 67: cat << \SHAR_EOF > Makefile ! 68: # ! 69: # Makefile for savenews ! 70: # ! 71: CFLAGS = -g ! 72: ! 73: savenews: savenews.c ! 74: ${CC} ${CFLAGS} savenews.c -o savenews ! 75: ! 76: clean: ! 77: rm -f savenews ! 78: ! 79: lint: ! 80: lint -hx savenews.c ! 81: SHAR_EOF ! 82: cat << \SHAR_EOF > savenews.c ! 83: /* ! 84: * savenews filename [filename ...] ! 85: * ! 86: * Savenews is a program designed to clean up and compact a ! 87: * usenet archive. It will take the filename(s) given to it as arguments ! 88: * and save them in a netnews archive (defined by SAVENEWS, default is ! 89: * /usr/spool/savenews). ! 90: * ! 91: * This program was set up to do two main things: ! 92: * ! 93: * 1) compact out the useless parts of the message, specifically the lines ! 94: * in the header that don't serve a useful purpose in an archive. This ! 95: * is done by removing all but the following header lines: From, Date, ! 96: * Newsgroups, Subject, and Message-ID, and seems to save an average of ! 97: * 500 bytes an article. ! 98: * ! 99: * 2) keep the quadratic nature of unix(TM AT&T Bell labs) directory searches ! 100: * from making your life miserable. Storing a raw archive of ! 101: * net.unix-wizards is a silly thing to do, for example. What I do is ! 102: * create a one level subdirectory set to keep any one directory from ! 103: * getting too large, but this program is currently set so that there ! 104: * are enough directories to keep the total number of files in any one ! 105: * directory below about 150 in the largest parts of my archive. The ! 106: * algorithm I use is abs(atoi(Message-ID)%HASHVAL)) with HASHVAL being ! 107: * prime. This quick and dirty hash gives you directories with the ! 108: * numbers 0 to HASHVAL-1, and about the same number of files in each ! 109: * given a random distribution of Message-ID numbers (not bad, in ! 110: * reality) ! 111: * ! 112: * The program will add the name of the file and the subject line of the ! 113: * article in a logfile in subdirectory LOGS, the filename being the ! 114: * newsgroup. ! 115: * ! 116: * As currently written, an article will be saved only to the first ! 117: * newsgroup in the Newsgroups header line. This means that something ! 118: * posted to 'net.source,net.flame' will end up in net.sources, but that ! 119: * somethine posted to 'net.flame,net.sources' will end up in net.flame. ! 120: * I consider this a feature. Others may disagree. ! 121: * ! 122: * If an article is saved that has a duplicate message-ID of one already ! 123: * in the archive, then it will be saved by adding the character '_' and ! 124: * some small integer needed to make the filename unique. You can then ! 125: * use ls or find to look for these and see if they are duplicates (and ! 126: * remove them) or if they are simply botches by some other site (it does ! 127: * happen, unfortunately). ! 128: * ! 129: * This program will do intelligent things if given a non-news article, ! 130: * such as nothing. Don't push it, though -- I haven't tried it on ! 131: * special devices, symbolic links, and other wierdies and it is likely ! 132: * to throw up on some of them since I didn`t feel like protecting someone ! 133: * from trying to archive /dev (if tar can consider this a feature, so can ! 134: * I...) ! 135: * ! 136: * This program uses the 4.2 Directory routines (libndir). If you don't ! 137: * run 4.2, get ahold of a copy of the compatibility library for your ! 138: * system and use it, or hack up do_dir and is_dir to get around it ! 139: * if you believe in messing around with primitive hacks (I LIKE libndir) ! 140: * ! 141: * General usage: every so often run the program with ! 142: * 'savenews /usr/spool/oldnews'. Look through /usr/spool/savenews ! 143: * for duplicated articles and remove them, and then copy all of the ! 144: * stuff to tape. Remove everything except the LOGS directory, so that ! 145: * people can use grep to look for things in the archive. It should be ! 146: * easy to get things back off of tape and make the archive useful this ! 147: * way. Thinking about it, if you can't use the archive, you might as well ! 148: * not have it, which is why this program got written (I needed something ! 149: * out of my archive, and it took me a week to find it). ! 150: * ! 151: * This program is designed to run under 2.10.2, but should work under any ! 152: * B news system. Anyone else is on their own. This is in ! 153: * the public domain by the kindness of my employer, national ! 154: * semiconductor, but neither I nor national make any guarantee that it ! 155: * will work, that we will support this program, or even admit that it ! 156: * exists. This is called a disclaimer, and means that if you use this ! 157: * program, you are on your own. It DOES, however, pass lint cleanly, which ! 158: * is more than I can say for most stuff posted to the net. Feel free to ! 159: * fix, break, enhance, change, or do anything to this program except ! 160: * claim it to be your own (unless, of course, you break it...). Passing ! 161: * enhancements back to me would be nice, too. ! 162: * ! 163: * chuq von rospach, national semiconductor (nsc!chuqui) ! 164: * ! 165: */ ! 166: ! 167: #include <stdio.h> ! 168: #include <sys/types.h> ! 169: #include <sys/stat.h> ! 170: #include <sys/dir.h> ! 171: #include <ctype.h> ! 172: ! 173: #define FALSE 0 ! 174: #define TRUE 1 ! 175: #define HASHVAL 37 /* hash value for sub-dirs. Prime number! */ ! 176: #define NUMDIRS 1024 /* number of dirs that can be pushed */ ! 177: #define SAVENEWS "/usr/spool/savenews" /* home of the archive */ ! 178: #define LOGFILE "LOGS" /* subdir in SAVENEWS to save logs in */ ! 179: #define JOBLOG "joblog" /* where log of this job is put */ ! 180: #define DIRMODE 0755 /* mkdir with this mode */ ! 181: #define COPYBUF 8192 /* block read/write buffer size */ ! 182: ! 183: char *Progname; /* name of the program for Eprintf */ ! 184: char line[BUFSIZ]; /* general purpose line buffer */ ! 185: ! 186: #define NUM_HEADERS 5 /* number of headers we are saving */ ! 187: #define GROUP_HEADER 1 /* where Newsgroup will be found */ ! 188: #define SUBJECT_HEADER 2 /* where Subject will be found */ ! 189: #define MESSAGE_HEADER 3 /* where Message-ID will be found */ ! 190: char header_data[NUM_HEADERS][BUFSIZ]; ! 191: char *headers[NUM_HEADERS] = ! 192: { ! 193: "From:", ! 194: "Newsgroups:", ! 195: "Subject:", ! 196: "Message-ID:", ! 197: "Date:" ! 198: }; ! 199: ! 200: long num_saved = 0; /* number of articles saved */ ! 201: FILE *logfp; /* file pointer to joblog file */ ! 202: ! 203: char *rindex(), *strcat(), *pop_dir(), *strcpy(), *strsave(), *index(); ! 204: ! 205: main(argc,argv) ! 206: int argc; ! 207: char *argv[]; ! 208: { ! 209: register int i; ! 210: char joblogfile[BUFSIZ]; ! 211: char *dirname; ! 212: ! 213: /* ! 214: * This removes and preceeding pathname so that ! 215: * anything printed out by Eprintf has just the ! 216: * program name and not where it came from ! 217: */ ! 218: if ((Progname = rindex(argv[0],'/')) == NULL) ! 219: Progname = argv[0]; ! 220: else ! 221: Progname++; ! 222: ! 223: if (argc == 1) { ! 224: fprintf(stderr,"Usage: %s file [file ...]\n",Progname); ! 225: exit(1); ! 226: } ! 227: ! 228: sprintf(joblogfile,"%s/%s",SAVENEWS,JOBLOG); ! 229: if ((logfp = fopen(joblogfile,"w")) == NULL) ! 230: fprintf(stderr,"Can't open %s, logging suspended\n",joblogfile); ! 231: ! 232: for (i = 1 ; i < argc; i++) { /* process each parameter */ ! 233: register int rc; ! 234: if ((rc = is_dir(argv[i])) == -1) ! 235: continue; ! 236: else if (rc == TRUE) ! 237: do_dir(argv[i]); ! 238: else ! 239: save_file(argv[i]); ! 240: } ! 241: while((dirname = pop_dir()) != NULL) { ! 242: do_dir(dirname); /* process whatever is left on dirstack */ ! 243: } ! 244: printf("Total articles saved was %d\n",num_saved); ! 245: exit(0); ! 246: } ! 247: ! 248: do_dir(dname) /* process a directory, push other directories on stack */ ! 249: /* to be handled recursively later */ ! 250: char *dname; ! 251: { ! 252: DIR *dirp; ! 253: struct direct *dp; ! 254: char fullname[BUFSIZ]; ! 255: ! 256: if ((dirp = opendir(dname)) == NULL) { ! 257: Eprintf("can't opendir %s\n",dname); ! 258: return; ! 259: } ! 260: ! 261: for (dp = readdir(dirp); dp != NULL; dp = readdir(dirp)) { ! 262: register int rc; ! 263: ! 264: if(dp->d_namlen == 2 && !strcmp(dp->d_name,"..") ! 265: || (dp->d_namlen == 1 && !strcmp(dp->d_name,"."))) ! 266: continue; /* skip . and .. */ ! 267: ! 268: sprintf(fullname,"%s/%s",dname,dp->d_name); ! 269: if((rc = is_dir(fullname)) == -1) ! 270: continue; ! 271: else if (rc == TRUE) ! 272: push_dir(fullname); ! 273: else ! 274: save_file(fullname); ! 275: } ! 276: closedir(dirp); ! 277: } ! 278: ! 279: is_dir(name) ! 280: char *name; ! 281: { ! 282: struct stat sbuf; ! 283: ! 284: if (stat(name,&sbuf) == -1) { ! 285: Eprintf("can't stat '%s'\n",name); ! 286: return(-1); ! 287: } ! 288: return((sbuf.st_mode & S_IFDIR) ? TRUE : FALSE); ! 289: } ! 290: ! 291: /* VARARGS */ ! 292: Eprintf(s1,s2,s3,s4,s5,s6,s7,s8,s9) ! 293: char *s1,*s2,*s3,*s4,*s5,*s6,*s7,*s8,*s9; ! 294: { ! 295: if (logfp == NULL) ! 296: return; ! 297: fprintf(logfp,"%s: ",Progname); ! 298: fprintf(logfp,s1,s2,s3,s4,s5,s6,s7,s8,s9); ! 299: fflush(logfp); ! 300: } ! 301: ! 302: /* ! 303: * quick and dirty stack routines. ! 304: * ! 305: * push_dir(name) char *name; ! 306: * stores the given string in the stack ! 307: * char *pop_dir() ! 308: * returns a string from the stack, or NULL if none. ! 309: */ ! 310: ! 311: static char *dirstack[NUMDIRS]; ! 312: static int lastdir = 0; ! 313: static char pop_name[BUFSIZ]; ! 314: ! 315: push_dir(name) ! 316: char *name; ! 317: { ! 318: if (lastdir >= NUMDIRS) { ! 319: Eprintf("push_dir overflow!\n"); ! 320: return; ! 321: } ! 322: dirstack[lastdir] = strsave(name); ! 323: if (dirstack[lastdir] == NULL) ! 324: { ! 325: Eprintf("malloc failed!\n"); ! 326: return; ! 327: } ! 328: lastdir++; ! 329: } ! 330: ! 331: char *pop_dir() ! 332: { ! 333: if(lastdir == 0) ! 334: return(NULL); ! 335: lastdir--; ! 336: strcpy(pop_name,dirstack[lastdir]); ! 337: dirstack[lastdir] = NULL; ! 338: free(dirstack[lastdir]); ! 339: return(pop_name); ! 340: } ! 341: ! 342: char *strsave(s) ! 343: char *s; ! 344: { ! 345: char *p, *malloc(); ! 346: ! 347: if ((p = malloc((unsigned)strlen(s)+1)) != NULL) ! 348: strcpy(p,s); ! 349: return(p); ! 350: } ! 351: ! 352: save_file(name) /* save the article in the archive */ ! 353: char *name; ! 354: { ! 355: FILE *fp, *ofp, *fopen(), *output_file(); ! 356: register int i, nc; ! 357: char diskbuf[COPYBUF]; ! 358: ! 359: Eprintf("saving '%s'\n",name); ! 360: if ((fp = fopen(name,"r")) == NULL) { ! 361: Eprintf("can't open\n"); ! 362: return; ! 363: } ! 364: ! 365: if ((fgets(line,BUFSIZ,fp) == NULL)) { ! 366: Eprintf("0 length file\n"); ! 367: fclose(fp); ! 368: return; ! 369: } ! 370: if (!start_header(line)) { ! 371: Eprintf("not a news article\n"); ! 372: fclose(fp); ! 373: return; ! 374: } ! 375: read_header(fp); ! 376: if ((ofp = output_file()) == NULL) { ! 377: Eprintf("Can't save\n"); ! 378: fclose(fp); ! 379: return; ! 380: } ! 381: ! 382: for (i = 0; i < NUM_HEADERS; i++) ! 383: fprintf(ofp,"%s\n",header_data[i]); ! 384: fputc('\n',ofp); ! 385: ! 386: while ((nc = fread(diskbuf,sizeof(char),COPYBUF,fp)) != 0) ! 387: fwrite(diskbuf,sizeof(char),nc,ofp); /* copy body of article */ ! 388: fclose(ofp); ! 389: fclose(fp); ! 390: num_saved++; ! 391: return; ! 392: } ! 393: ! 394: start_header(s) /* see if this is the start of a news article */ ! 395: char *s; ! 396: { ! 397: /* ! 398: * If this is coming from B news, the first line will 'always' be ! 399: * Relay-Version (at least, on my system). Your mileage my vary. ! 400: */ ! 401: if (!strncmp(s,"Relay-Version:",14)) ! 402: return(TRUE); ! 403: /* ! 404: * If you are copying a section of archive already archived by ! 405: * sendnews, then the first line will be From (unless you changed ! 406: * the headers data structure, then its up to you...) ! 407: */ ! 408: if (!strncmp(s,"From:",5)) ! 409: return(TRUE); ! 410: return(FALSE); ! 411: } ! 412: ! 413: /* ! 414: * By the time we get here, the first line will already be read in and ! 415: * checked by start_header(). If we are re-copying a savenews archive ! 416: * (which happens when you decide to play with HASHVAL, trust me) then ! 417: * we need to save the From line, so we can't just throw it away. Hence ! 418: * the funky looking do-while setup instead of something a bit more ! 419: * straightforward ! 420: */ ! 421: read_header(fp) ! 422: FILE *fp; ! 423: { ! 424: register int i; ! 425: ! 426: for (i = 0; i < NUM_HEADERS; i++) ! 427: header_data[i][0] = '\0'; /* remove last articles data */ ! 428: ! 429: do { ! 430: char *cp; ! 431: ! 432: if (line[0] == '\n') /* always be a blank line after the header */ ! 433: return; ! 434: ! 435: for (i = 0 ; i < NUM_HEADERS; i++) { ! 436: if (!strncmp(headers[i],line,strlen(headers[i]))) { ! 437: strcpy(header_data[i],line); ! 438: if (cp = index(header_data[i],'\n')) ! 439: *cp = '\0'; /* eat newlines */ ! 440: } ! 441: } ! 442: } while (fgets(line,BUFSIZ,fp) != NULL); ! 443: } ! 444: ! 445: FILE *output_file() /* generate the name in the archive */ ! 446: { ! 447: int hashval, copy = 0; ! 448: FILE *fp, *fopen(); ! 449: char *p, newsgroup[BUFSIZ], message_id[BUFSIZ]; ! 450: char shortname[BUFSIZ], filename[BUFSIZ], filename2[BUFSIZ]; ! 451: ! 452: /* get the first newsgroup */ ! 453: p = index(header_data[GROUP_HEADER],':'); /* move past Newsgroups */ ! 454: if (!p) { ! 455: Eprintf("Invalid newsgroups\n"); ! 456: return(NULL); ! 457: } ! 458: p++; /* skip the colon */ ! 459: while (isspace(*p)) ! 460: p++; /* skip whitespace */ ! 461: strcpy(newsgroup,p); ! 462: if (p = index(newsgroup,',')) ! 463: *p= '\0'; /* newsgroup now only has one name in it */ ! 464: ! 465: /* get the message-id */ ! 466: p = index(header_data[MESSAGE_HEADER],':'); ! 467: if (!p) { ! 468: Eprintf("Invalid message-id\n"); ! 469: return(NULL); ! 470: } ! 471: p++; /* skip the colon */ ! 472: while (isspace(*p)) ! 473: p++; /* skip whitespace */ ! 474: if (*p == '<' || *p == '(') ! 475: p++; ! 476: if (*p == '-') /* make negative article id numbers positive (hack) */ ! 477: p++; ! 478: strcpy(message_id,p); ! 479: if (p = index(message_id,'.')) /* trim off the .UUCP if any */ ! 480: *p = '\0'; ! 481: else if (p = index(message_id,'>')) /* or get the closing bracket */ ! 482: *p = '\0'; ! 483: else if (p = index(message_id,')')) /* or get the closing paren */ ! 484: *p = '\0'; ! 485: if (p = index(message_id,'@')) /* change nnn@site */ ! 486: *p = '.'; /* to nnn.site */ ! 487: ! 488: /* generate the hash value for the subdirectory */ ! 489: hashval = atoi(message_id) % HASHVAL; ! 490: ! 491: /* setup the filename to save to */ ! 492: sprintf(shortname,"%s/%d/%s",newsgroup,hashval,message_id); ! 493: sprintf(filename,"%s/%s",SAVENEWS,shortname); ! 494: while (exists(filename)) { /* make it unique if neccessary */ ! 495: ! 496: sprintf(shortname,"%s/%d/%s_%d",newsgroup,hashval,message_id,++copy); ! 497: sprintf(filename,"%s/%s",SAVENEWS,shortname); ! 498: } ! 499: ! 500: strcpy(filename2,filename); /* must chop off the filename */ ! 501: if (p = rindex(filename2,'/')) /* since we don't want to */ ! 502: *p = '\0'; /* to makeparents */ ! 503: makeparents(filename2); ! 504: ! 505: if ((fp = fopen(filename,"w")) == NULL) { ! 506: Eprintf("Can't open %s for output\n",filename); ! 507: return(NULL); ! 508: } ! 509: log(newsgroup,shortname); ! 510: return(fp); ! 511: } ! 512: ! 513: exists(name) ! 514: char *name; ! 515: { ! 516: struct stat sbuf; ! 517: ! 518: if (stat(name,&sbuf) == -1) { ! 519: return(FALSE); ! 520: } ! 521: return(TRUE); ! 522: } ! 523: ! 524: makeparents(name) /* recursively make parent directories */ ! 525: char *name; ! 526: { ! 527: char *p, buf[BUFSIZ]; ! 528: ! 529: if (exists(name)) ! 530: return; ! 531: strcpy(buf,name); ! 532: if (!(p = rindex(buf,'/'))) { ! 533: Eprintf("makeparents failed!\n"); ! 534: return; ! 535: } ! 536: *p = '\0'; ! 537: makeparents(buf); ! 538: mkdir(name,DIRMODE); ! 539: } ! 540: ! 541: log(group,name) /* write to the logfile */ ! 542: char *group, *name; ! 543: { ! 544: char *subject, logfile[BUFSIZ]; ! 545: FILE *ofp, *fopen(); ! 546: ! 547: /* get the subject */ ! 548: subject = index(header_data[SUBJECT_HEADER],':'); ! 549: if (!subject) { ! 550: Eprintf("Invalid subject, no log entry\n"); ! 551: return; ! 552: } ! 553: subject++; /* skip the colon */ ! 554: while (isspace(*subject)) ! 555: subject++; /* skip whitespace */ ! 556: ! 557: /* generate the place where it goes */ ! 558: sprintf(logfile,"%s/%s",SAVENEWS,LOGFILE); ! 559: makeparents(logfile); ! 560: strcat(logfile,"/"); ! 561: strcat(logfile,group); ! 562: ! 563: if ((ofp = fopen(logfile,"a")) == NULL) ! 564: { ! 565: Eprintf("open failed on %s\n",logfile); ! 566: return; ! 567: } ! 568: fprintf(ofp,"%s\t%s\n", name, subject); ! 569: fclose(ofp); ! 570: } ! 571: ! 572: SHAR_EOF ! 573: # End of shell archive ! 574: exit 0 ! 575: -- ! 576: :From the misfiring synapses of: Chuq Von Rospach ! 577: {cbosgd,fortune,hplabs,ihnp4,seismo}!nsc!chuqui [email protected] ! 578: ! 579: The offices were very nice, and the clients were only raping the land, and ! 580: then, of course, there was the money... ! 581: ! 582:
This archive runs on limited infrastructure. Preserving old code on modern bandwidth. Automated agents are requested to crawl responsibly.