Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 244 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 5.2395. Min Range: -5.2395

Max Act: 5.2395. Min Act: -0.1700

Data Index: 118548 (C4 (Web Text))

Max Activating Token Index: 559

Text #1

Max Range: 5.2395. Min Range: -5.2395

Max Act: 5.2395. Min Act: -0.1700

Data Index: 26817 (C4 (Web Text))

Max Activating Token Index: 992

Text #2

Max Range: 5.2395. Min Range: -5.2395

Max Act: 5.1145. Min Act: -0.1700

Data Index: 935638 (C4 (Web Text))

Max Activating Token Index: 929

Text #3

Max Range: 5.2395. Min Range: -5.2395

Max Act: 5.2082. Min Act: -0.1700

Data Index: 1045267 (C4 (Web Text))

Max Activating Token Index: 477

Text #4

Max Range: 5.2395. Min Range: -5.2395

Max Act: 5.0832. Min Act: -0.1700

Data Index: 296263 (C4 (Web Text))

Max Activating Token Index: 391

Text #5

Max Range: 5.2395. Min Range: -5.2395

Max Act: 5.1457. Min Act: -0.1700

Data Index: 448031 (C4 (Web Text))

Max Activating Token Index: 345

Text #6

Max Range: 5.2395. Min Range: -5.2395

Max Act: 5.1770. Min Act: -0.1700

Data Index: 969790 (C4 (Web Text))

Max Activating Token Index: 342

Text #7

Max Range: 5.2395. Min Range: -5.2395

Max Act: 5.1457. Min Act: -0.1700

Data Index: 1174417 (C4 (Web Text))

Max Activating Token Index: 848

Text #8

Max Range: 5.2395. Min Range: -5.2395

Max Act: 5.0832. Min Act: -0.1700

Data Index: 857064 (C4 (Web Text))

Max Activating Token Index: 297

Text #9

Max Range: 5.2395. Min Range: -5.2395

Max Act: 5.1145. Min Act: -0.1700

Data Index: 928826 (C4 (Web Text))

Max Activating Token Index: 412

Text #10

Max Range: 5.2395. Min Range: -5.2395

Max Act: 4.9582. Min Act: -0.1700

Data Index: 1151233 (C4 (Web Text))

Max Activating Token Index: 455

Text #11

Max Range: 5.2395. Min Range: -5.2395

Max Act: 5.0207. Min Act: -0.1700

Data Index: 256758 (C4 (Web Text))

Max Activating Token Index: 31

Text #12

Max Range: 5.2395. Min Range: -5.2395

Max Act: 5.0207. Min Act: -0.1700

Data Index: 442955 (C4 (Web Text))

Max Activating Token Index: 379

Text #13

Max Range: 5.2395. Min Range: -5.2395

Max Act: 4.9895. Min Act: -0.1700

Data Index: 474249 (C4 (Web Text))

Max Activating Token Index: 94

Text #14

Max Range: 5.2395. Min Range: -5.2395

Max Act: 5.0520. Min Act: -0.1700

Data Index: 803363 (C4 (Web Text))

Max Activating Token Index: 764

Text #15

Max Range: 5.2395. Min Range: -5.2395

Max Act: 5.1145. Min Act: -0.1700

Data Index: 136117 (C4 (Web Text))

Max Activating Token Index: 533

Text #16

Max Range: 5.2395. Min Range: -5.2395

Max Act: 4.9582. Min Act: -0.1700

Data Index: 773873 (C4 (Web Text))

Max Activating Token Index: 476

Text #17

Max Range: 5.2395. Min Range: -5.2395

Max Act: 4.9270. Min Act: -0.1700

Data Index: 410417 (C4 (Web Text))

Max Activating Token Index: 969

Text #18

Max Range: 5.2395. Min Range: -5.2395

Max Act: 4.9895. Min Act: -0.1700

Data Index: 269058 (C4 (Web Text))

Max Activating Token Index: 738

Text #19

Max Range: 5.2395. Min Range: -5.2395

Max Act: 4.9270. Min Act: -0.1700

Data Index: 1043033 (C4 (Web Text))

Max Activating Token Index: 669