Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3657 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.5500. Min Range: -2.5500

Max Act: 2.5500. Min Act: -0.0000

Data Index: 788495 (C4 (Web Text))

Max Activating Token Index: 739

Text #1

Max Range: 2.5500. Min Range: -2.5500

Max Act: 2.4634. Min Act: -0.0000

Data Index: 828948 (C4 (Web Text))

Max Activating Token Index: 617

Text #2

Max Range: 2.5500. Min Range: -2.5500

Max Act: 2.3688. Min Act: -0.0000

Data Index: 450552 (C4 (Web Text))

Max Activating Token Index: 960

Text #3

Max Range: 2.5500. Min Range: -2.5500

Max Act: 2.3260. Min Act: -0.0000

Data Index: 226025 (C4 (Web Text))

Max Activating Token Index: 431

Text #4

Max Range: 2.5500. Min Range: -2.5500

Max Act: 2.3046. Min Act: -0.0000

Data Index: 1236138 (C4 (Web Text))

Max Activating Token Index: 264

Text #5

Max Range: 2.5500. Min Range: -2.5500

Max Act: 2.0489. Min Act: -0.0000

Data Index: 600516 (C4 (Web Text))

Max Activating Token Index: 99

Text #6

Max Range: 2.5500. Min Range: -2.5500

Max Act: 2.1221. Min Act: -0.0000

Data Index: 1175212 (C4 (Web Text))

Max Activating Token Index: 516

Text #7

Max Range: 2.5500. Min Range: -2.5500

Max Act: 2.0409. Min Act: -0.0000

Data Index: 195974 (C4 (Web Text))

Max Activating Token Index: 130

Text #8

Max Range: 2.5500. Min Range: -2.5500

Max Act: 1.7754. Min Act: -0.0000

Data Index: 883322 (C4 (Web Text))

Max Activating Token Index: 657

Text #9

Max Range: 2.5500. Min Range: -2.5500

Max Act: 1.8417. Min Act: -0.0000

Data Index: 885129 (C4 (Web Text))

Max Activating Token Index: 736

Text #10

Max Range: 2.5500. Min Range: -2.5500

Max Act: 1.9243. Min Act: -0.0000

Data Index: 622799 (C4 (Web Text))

Max Activating Token Index: 317

Text #11

Max Range: 2.5500. Min Range: -2.5500

Max Act: 1.7947. Min Act: -0.0000

Data Index: 259978 (C4 (Web Text))

Max Activating Token Index: 629

Text #12

Max Range: 2.5500. Min Range: -2.5500

Max Act: 1.6923. Min Act: -0.0000

Data Index: 567884 (C4 (Web Text))

Max Activating Token Index: 762

Text #13

Max Range: 2.5500. Min Range: -2.5500

Max Act: 1.7751. Min Act: -0.0000

Data Index: 55458 (C4 (Web Text))

Max Activating Token Index: 1012

Text #14

Max Range: 2.5500. Min Range: -2.5500

Max Act: 1.8684. Min Act: -0.0000

Data Index: 1045410 (C4 (Web Text))

Max Activating Token Index: 575

Text #15

Max Range: 2.5500. Min Range: -2.5500

Max Act: 1.6269. Min Act: -0.0000

Data Index: 1350296 (C4 (Web Text))

Max Activating Token Index: 391

Text #16

Max Range: 2.5500. Min Range: -2.5500

Max Act: 1.7014. Min Act: -0.0000

Data Index: 855900 (C4 (Web Text))

Max Activating Token Index: 27

Text #17

Max Range: 2.5500. Min Range: -2.5500

Max Act: 1.6956. Min Act: -0.0000

Data Index: 590916 (C4 (Web Text))

Max Activating Token Index: 260

Text #18

Max Range: 2.5500. Min Range: -2.5500

Max Act: 1.6331. Min Act: -0.0000

Data Index: 1229648 (C4 (Web Text))

Max Activating Token Index: 28

Text #19

Max Range: 2.5500. Min Range: -2.5500

Max Act: 1.6262. Min Act: -0.0000

Data Index: 23729 (C4 (Web Text))

Max Activating Token Index: 730