Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4462 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.2999. Min Range: -2.2999

Max Act: 2.1806. Min Act: -0.0000

Data Index: 827446 (C4 (Web Text))

Max Activating Token Index: 985

Text #1

Max Range: 2.2999. Min Range: -2.2999

Max Act: 2.1599. Min Act: -0.0000

Data Index: 259314 (C4 (Web Text))

Max Activating Token Index: 418

Text #2

Max Range: 2.2999. Min Range: -2.2999

Max Act: 2.1559. Min Act: -0.0000

Data Index: 854019 (C4 (Web Text))

Max Activating Token Index: 412

Text #3

Max Range: 2.2999. Min Range: -2.2999

Max Act: 2.0886. Min Act: -0.0000

Data Index: 1212526 (C4 (Web Text))

Max Activating Token Index: 695

Text #4

Max Range: 2.2999. Min Range: -2.2999

Max Act: 1.9872. Min Act: -0.0000

Data Index: 1304303 (C4 (Web Text))

Max Activating Token Index: 999

Text #5

Max Range: 2.2999. Min Range: -2.2999

Max Act: 2.2999. Min Act: -0.0000

Data Index: 417515 (C4 (Web Text))

Max Activating Token Index: 889

Text #6

Max Range: 2.2999. Min Range: -2.2999

Max Act: 2.1964. Min Act: -0.0000

Data Index: 22883 (C4 (Web Text))

Max Activating Token Index: 673

Text #7

Max Range: 2.2999. Min Range: -2.2999

Max Act: 2.2711. Min Act: -0.0000

Data Index: 190474 (C4 (Web Text))

Max Activating Token Index: 1015

Text #8

Max Range: 2.2999. Min Range: -2.2999

Max Act: 2.2624. Min Act: -0.0000

Data Index: 1336874 (C4 (Web Text))

Max Activating Token Index: 732

Text #9

Max Range: 2.2999. Min Range: -2.2999

Max Act: 2.1206. Min Act: -0.0000

Data Index: 1093169 (C4 (Web Text))

Max Activating Token Index: 626

Text #10

Max Range: 2.2999. Min Range: -2.2999

Max Act: 2.1852. Min Act: -0.0000

Data Index: 1108473 (C4 (Web Text))

Max Activating Token Index: 430

Text #11

Max Range: 2.2999. Min Range: -2.2999

Max Act: 1.9767. Min Act: -0.0000

Data Index: 949793 (C4 (Web Text))

Max Activating Token Index: 229

Text #12

Max Range: 2.2999. Min Range: -2.2999

Max Act: 2.0224. Min Act: -0.0000

Data Index: 560511 (C4 (Web Text))

Max Activating Token Index: 779

Text #13

Max Range: 2.2999. Min Range: -2.2999

Max Act: 2.0248. Min Act: -0.0000

Data Index: 393368 (C4 (Web Text))

Max Activating Token Index: 79

Text #14

Max Range: 2.2999. Min Range: -2.2999

Max Act: 2.0360. Min Act: -0.0000

Data Index: 12035 (C4 (Web Text))

Max Activating Token Index: 977

Text #15

Max Range: 2.2999. Min Range: -2.2999

Max Act: 2.0314. Min Act: -0.0000

Data Index: 132269 (C4 (Web Text))

Max Activating Token Index: 144

Text #16

Max Range: 2.2999. Min Range: -2.2999

Max Act: 2.0172. Min Act: -0.0000

Data Index: 155677 (C4 (Web Text))

Max Activating Token Index: 894

Text #17

Max Range: 2.2999. Min Range: -2.2999

Max Act: 1.9104. Min Act: -0.0000

Data Index: 1183034 (C4 (Web Text))

Max Activating Token Index: 817

Text #18

Max Range: 2.2999. Min Range: -2.2999

Max Act: 1.9092. Min Act: -0.0000

Data Index: 372185 (C4 (Web Text))

Max Activating Token Index: 112

Text #19

Max Range: 2.2999. Min Range: -2.2999

Max Act: 1.9807. Min Act: -0.0000

Data Index: 799906 (C4 (Web Text))

Max Activating Token Index: 194