Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 381 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.4826. Min Range: -0.4826

Max Act: 0.4826. Min Act: -0.0000

Data Index: 674525 (C4 (Web Text))

Max Activating Token Index: 395

Text #1

Max Range: 0.4826. Min Range: -0.4826

Max Act: 0.4379. Min Act: -0.0000

Data Index: 151194 (C4 (Web Text))

Max Activating Token Index: 949

Text #2

Max Range: 0.4826. Min Range: -0.4826

Max Act: 0.3801. Min Act: -0.0000

Data Index: 879618 (C4 (Web Text))

Max Activating Token Index: 855

Text #3

Max Range: 0.4826. Min Range: -0.4826

Max Act: 0.3722. Min Act: -0.0001

Data Index: 1115537 (C4 (Web Text))

Max Activating Token Index: 178

Text #4

Max Range: 0.4826. Min Range: -0.4826

Max Act: 0.3476. Min Act: -0.0000

Data Index: 407358 (C4 (Web Text))

Max Activating Token Index: 408

Text #5

Max Range: 0.4826. Min Range: -0.4826

Max Act: 0.3403. Min Act: 0.0001

Data Index: 1264853 (C4 (Web Text))

Max Activating Token Index: 276

Text #6

Max Range: 0.4826. Min Range: -0.4826

Max Act: 0.3290. Min Act: 0.0000

Data Index: 40785 (C4 (Web Text))

Max Activating Token Index: 984

Text #7

Max Range: 0.4826. Min Range: -0.4826

Max Act: 0.3504. Min Act: -0.0000

Data Index: 1263836 (C4 (Web Text))

Max Activating Token Index: 802

Text #8

Max Range: 0.4826. Min Range: -0.4826

Max Act: 0.3227. Min Act: -0.0001

Data Index: 77573 (C4 (Web Text))

Max Activating Token Index: 503

Text #9

Max Range: 0.4826. Min Range: -0.4826

Max Act: 0.3212. Min Act: -0.0000

Data Index: 1324851 (C4 (Web Text))

Max Activating Token Index: 705

Text #10

Max Range: 0.4826. Min Range: -0.4826

Max Act: 0.3421. Min Act: -0.0000

Data Index: 343454 (C4 (Web Text))

Max Activating Token Index: 183

Text #11

Max Range: 0.4826. Min Range: -0.4826

Max Act: 0.3423. Min Act: -0.0001

Data Index: 333160 (C4 (Web Text))

Max Activating Token Index: 200

Text #12

Max Range: 0.4826. Min Range: -0.4826

Max Act: 0.3142. Min Act: -0.0001

Data Index: 499625 (C4 (Web Text))

Max Activating Token Index: 350

Text #13

Max Range: 0.4826. Min Range: -0.4826

Max Act: 0.3218. Min Act: -0.0000

Data Index: 762353 (C4 (Web Text))

Max Activating Token Index: 708

Text #14

Max Range: 0.4826. Min Range: -0.4826

Max Act: 0.3388. Min Act: -0.0000

Data Index: 124745 (C4 (Web Text))

Max Activating Token Index: 149

Text #15

Max Range: 0.4826. Min Range: -0.4826

Max Act: 0.3245. Min Act: 0.0000

Data Index: 880456 (C4 (Web Text))

Max Activating Token Index: 489

Text #16

Max Range: 0.4826. Min Range: -0.4826

Max Act: 0.3240. Min Act: -0.0000

Data Index: 617131 (C4 (Web Text))

Max Activating Token Index: 130

Text #17

Max Range: 0.4826. Min Range: -0.4826

Max Act: 0.3214. Min Act: -0.0000

Data Index: 394376 (C4 (Web Text))

Max Activating Token Index: 808

Text #18

Max Range: 0.4826. Min Range: -0.4826

Max Act: 0.3096. Min Act: -0.0000

Data Index: 453938 (C4 (Web Text))

Max Activating Token Index: 784

Text #19

Max Range: 0.4826. Min Range: -0.4826

Max Act: 0.3167. Min Act: -0.0000

Data Index: 174576 (C4 (Web Text))

Max Activating Token Index: 916