Your are here : Home / Data Processing Division

DATA PROCESSING DIVISION

DATA PROCESSING ACTIVITIES PERFORMED BY THE D.P. DIVISION        


     The Office of the Registrar General, India (ORGI) has always been on adoption of latest available technology up-gradation of data processing capabilities and capacity within the organization and particularly in Data Processing Division. The entire computer based processing of census project is handled by Data Processing Division, New Delhi


     In 1961 census, a modest beginning was made with the adoption of “Unit record” machines. Prior to 1961 censuses, the data collection, data entry and processing were done manually. In 1961 census, the Hand Punching machines (inserting one card at a time) using 80 column (Hollirith) punch cards were used for converting data into machine readable form. The processing was done on the sample (data) selected from the entire data. Around 70 ORGI officers were involved in the data entry, programming & machine operations. The data schedules were coded at various regional tabulation centres throughout India before sending the schedules for data entry. Reproducer was used to duplicate the entire set of punched cards. Data Processing was done by using Verifiers, sorter machines. “Serial Rolling Total Tabulator cum printer (SRTT)” was used for tabulation and printing of census tables.  Data backup system was not available at that time.


     In 1981 Census, first time in Indian census, the data entry activities were decentralised and 15 data centres were setup in major states including at one at headquarters, ORGI. Each centre was allotted more than one state (adjoining states) to convert paper-based information into machine-readable form using “key to disk” machines provided by GCS, ECIL & ICT. The data processing was done using HP1000, CD-Cyber 730 & NEC-1000 computer   systems at NIC, New Delhi and Regional Computer Centre (RCC), Chandigarh. The in-house data processing facilities were not available in ORGI.  The data schedules were coded at various regional tabulation centres throughout India before sending the schedules for data entry  to allotted one of the 15 data centres. Around 1200 ORGI officers including officers at 15 data centres, were engaged for the entire census data processing activities. All the required software (for data validation, editing, processing and tabulation) were developed by the officers of Data Processing Division, ORGI, Headquarters.


     During 1991 Census, a drastic change in the data processing activities in ORGI was taken place. ORGI had setup its own computing facility by installing Medha-930 main frame system at DP Division, Pushpa Bhawan. Dump terminals, under Unix operating system, connected to the servers at 15 data centres, were used for data entry.  Data movement between various data centers and Computer centre at Delhi was done through magnetic tapes. The data schedules were coded at various regional tabulation centres (about 163) throughout India before sending the schedules to the allotted one of the 15 data centres for data entry. Master data files creation, data editing on basic fields and lower level tabulations were processed at 4 Regional Processing Centres namely Delhi, Bhopal (Madhya Pradesh), Bhubaneshwar (Orissa) and Chennai (Tamil Nadu). Major editing considering all the fields, processing and generation of all the tables at various levels were taken up in DP Division, ORGI, HQr. All the required software (for data validation, editing, processing and tabulation) were developed by the officers of Data Processing Division, ORGI, HQr. First time, in the 1991 census, DP Division generated the camera ready copies of the tabulations in Hindi as well as in English for publication. The cent percent data was captured. The processing was done on the cent percent data of workers & SC/ST. Only 10 per cent records were processed for other tabulations. Around 1200 ORGI officers including officers at 15 data centres were engaged for the entire census data processing activities. The software were developed in-house by DP Division officers.


     At 2001 census, large scale hardware up-gradation had been made at 15 Data Centres and in DP Division, New Delhi. This was required for using the latest “Automatic Form Processing Technology using “Intelligent Character Recognition (ICR)” technology. The OMR/OCR/ICR technologies were evaluated and ICR was found to be acceptable being the viable solution for census data processing activities. IT resources were created at 15 data centres by installing 45 NT servers, 1060 PIII PCs, 25 High speed heavy duty duplex scanners (Kodak) and used backup devices like ZIP SLR & DLT drives. About 1200 technical officers and about 500 contractual operators were engaged for electronically data capturing at 15 data centres and processing of census data at DP Division, HQr., ORGI, New Delhi. The paper schedules were transported to the nearest allotted data center (resulted in reduced transportation cost) for scanning and data file creation activities. The scanning activities were carried out on 24x7x30 basis. The technology enabled processing of cent percent data, that is, more than one billion records (228 million paper forms were scanned) for the first time in census history. The scanned images were archived for permanent storage. Image enabled Computer Assisted Coding (CAC) was applied for certain fields. The use of ICR technology after scanning the schedules and use of CAC (eliminated setting up of Regional Tabulation Centres for manual coding) saved the Government lot of expenses incurred in the past on setting up of Regional Tabulation Offices. The services of a System Integrator were utilised at 15 Data Centres for scanning operations and data file creation. The use of technology has helped to enhance the skills of EDP officers of ORGI. The DP Division officers have developed the data processing software in-house.


The DP Division has assisted various Technical Divisions of ORGI for data processing projects, such as:  


     Technology Options for Census 2011 Following Server Hardware and storage has been installed in D.P. Division and in 15 Nos Data Centers in India for data processing activities- :  

    1.HP ProLiant DL380G6 Quad Core Base Servers  

     2.HP Storage Works EVA6400, Capacity – 10/100 TB  

     3.Kodak High Speed Scanners  

IT resources were created at 15 data centres by installing 36 Nos Microsoft Windows 2008R2 servers, SQL Server -2008, Windows 7 Professional Clients, High speed heavy duty duplex scanners (Kodak) and backup HP Storage Works EVA6400, Capacity – 10/100 TB. In addition to above Hardware & Software an Intelligent Character Recognition (ICR) software eFlow4.5 has also been installed to meet the challenges of Census-2011. About 1200 technical officers and about 500 contractual operators were engaged for electronically data capturing at 15 data centres and processing of census data at DP Division, New Delhi.
About 1200 technical officers and about 500 contractual operators were engaged for electronically data capturing at 15 data centres and processing of census data at DP Division, New Delhi. The use of ICR technology after scanning the schedules and use of Computer Assisted Coding (CAC) saves a lot of Government expenditure incurred in the past on setting up of Regional Tabulation Offices. The services of a System Integrator were utilized at 15 Data Centres for scanning operations and data file creation.
All the required softwares & MIS tools are developed by the officers of Data Processing Division, ORGI, New Delhi.