Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 528 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.9128. Min Range: -2.9128

Max Act: 2.9128. Min Act: -0.0000

Data Index: 938379 (C4 (Web Text))

Max Activating Token Index: 41

Text #1

Max Range: 2.9128. Min Range: -2.9128

Max Act: 2.7265. Min Act: -0.0000

Data Index: 1012394 (C4 (Web Text))

Max Activating Token Index: 281

Text #2

Max Range: 2.9128. Min Range: -2.9128

Max Act: 2.5022. Min Act: -0.0000

Data Index: 694314 (C4 (Web Text))

Max Activating Token Index: 35

Text #3

Max Range: 2.9128. Min Range: -2.9128

Max Act: 2.2399. Min Act: -0.0000

Data Index: 1374318 (Python Code)

Max Activating Token Index: 691

Text #4

Max Range: 2.9128. Min Range: -2.9128

Max Act: 2.5334. Min Act: -0.0000

Data Index: 1427199 (Python Code)

Max Activating Token Index: 830

Text #5

Max Range: 2.9128. Min Range: -2.9128

Max Act: 2.2491. Min Act: -0.0000

Data Index: 1442679 (Python Code)

Max Activating Token Index: 535

Text #6

Max Range: 2.9128. Min Range: -2.9128

Max Act: 2.3220. Min Act: -0.0000

Data Index: 1046536 (C4 (Web Text))

Max Activating Token Index: 315

Text #7

Max Range: 2.9128. Min Range: -2.9128

Max Act: 2.4505. Min Act: -0.0000

Data Index: 374901 (C4 (Web Text))

Max Activating Token Index: 237

Text #8

Max Range: 2.9128. Min Range: -2.9128

Max Act: 1.9702. Min Act: -0.0000

Data Index: 583771 (C4 (Web Text))

Max Activating Token Index: 186

Text #9

Max Range: 2.9128. Min Range: -2.9128

Max Act: 2.1846. Min Act: -0.0000

Data Index: 860491 (C4 (Web Text))

Max Activating Token Index: 223

Text #10

Max Range: 2.9128. Min Range: -2.9128

Max Act: 2.0580. Min Act: -0.0000

Data Index: 1634100 (Python Code)

Max Activating Token Index: 210

Text #11

Max Range: 2.9128. Min Range: -2.9128

Max Act: 2.0312. Min Act: -0.0000

Data Index: 1298358 (C4 (Web Text))

Max Activating Token Index: 352

Text #12

Max Range: 2.9128. Min Range: -2.9128

Max Act: 2.0038. Min Act: -0.0000

Data Index: 270945 (C4 (Web Text))

Max Activating Token Index: 390

Text #13

Max Range: 2.9128. Min Range: -2.9128

Max Act: 2.1000. Min Act: -0.0000

Data Index: 1316769 (C4 (Web Text))

Max Activating Token Index: 419

Text #14

Max Range: 2.9128. Min Range: -2.9128

Max Act: 2.0765. Min Act: -0.0000

Data Index: 44344 (C4 (Web Text))

Max Activating Token Index: 170

Text #15

Max Range: 2.9128. Min Range: -2.9128

Max Act: 1.7756. Min Act: -0.0000

Data Index: 1517969 (Python Code)

Max Activating Token Index: 916

Text #16

Max Range: 2.9128. Min Range: -2.9128

Max Act: 1.9459. Min Act: -0.0000

Data Index: 224781 (C4 (Web Text))

Max Activating Token Index: 264

Text #17

Max Range: 2.9128. Min Range: -2.9128

Max Act: 2.0032. Min Act: -0.0000

Data Index: 519618 (C4 (Web Text))

Max Activating Token Index: 240

Text #18

Max Range: 2.9128. Min Range: -2.9128

Max Act: 1.9942. Min Act: -0.0000

Data Index: 434809 (C4 (Web Text))

Max Activating Token Index: 120

Text #19

Max Range: 2.9128. Min Range: -2.9128

Max Act: 1.9764. Min Act: -0.0000

Data Index: 1400971 (Python Code)

Max Activating Token Index: 651