Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5396 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.2090. Min Range: -0.2090

Max Act: 0.1968. Min Act: -0.0000

Data Index: 428417 (C4 (Web Text))

Max Activating Token Index: 319

Text #1

Max Range: 0.2090. Min Range: -0.2090

Max Act: 0.2041. Min Act: -0.0000

Data Index: 1105873 (C4 (Web Text))

Max Activating Token Index: 508

Text #2

Max Range: 0.2090. Min Range: -0.2090

Max Act: 0.2024. Min Act: -0.0000

Data Index: 471308 (C4 (Web Text))

Max Activating Token Index: 440

Text #3

Max Range: 0.2090. Min Range: -0.2090

Max Act: 0.2071. Min Act: -0.0000

Data Index: 215393 (C4 (Web Text))

Max Activating Token Index: 212

Text #4

Max Range: 0.2090. Min Range: -0.2090

Max Act: 0.1941. Min Act: -0.0000

Data Index: 1288647 (C4 (Web Text))

Max Activating Token Index: 199

Text #5

Max Range: 0.2090. Min Range: -0.2090

Max Act: 0.2045. Min Act: -0.0000

Data Index: 378329 (C4 (Web Text))

Max Activating Token Index: 127

Text #6

Max Range: 0.2090. Min Range: -0.2090

Max Act: 0.1841. Min Act: -0.0000

Data Index: 165028 (C4 (Web Text))

Max Activating Token Index: 683

Text #7

Max Range: 0.2090. Min Range: -0.2090

Max Act: 0.1976. Min Act: -0.0000

Data Index: 1320685 (C4 (Web Text))

Max Activating Token Index: 154

Text #8

Max Range: 0.2090. Min Range: -0.2090

Max Act: 0.2090. Min Act: -0.0000

Data Index: 412818 (C4 (Web Text))

Max Activating Token Index: 279

Text #9

Max Range: 0.2090. Min Range: -0.2090

Max Act: 0.2036. Min Act: -0.0000

Data Index: 472631 (C4 (Web Text))

Max Activating Token Index: 247

Text #10

Max Range: 0.2090. Min Range: -0.2090

Max Act: 0.1798. Min Act: -0.0000

Data Index: 193291 (C4 (Web Text))

Max Activating Token Index: 1008

Text #11

Max Range: 0.2090. Min Range: -0.2090

Max Act: 0.1983. Min Act: -0.0000

Data Index: 3463 (C4 (Web Text))

Max Activating Token Index: 479

Text #12

Max Range: 0.2090. Min Range: -0.2090

Max Act: 0.1980. Min Act: -0.0000

Data Index: 1035030 (C4 (Web Text))

Max Activating Token Index: 384

Text #13

Max Range: 0.2090. Min Range: -0.2090

Max Act: 0.1914. Min Act: -0.0000

Data Index: 200976 (C4 (Web Text))

Max Activating Token Index: 605

Text #14

Max Range: 0.2090. Min Range: -0.2090

Max Act: 0.1727. Min Act: -0.0000

Data Index: 452537 (C4 (Web Text))

Max Activating Token Index: 755

Text #15

Max Range: 0.2090. Min Range: -0.2090

Max Act: 0.1974. Min Act: -0.0000

Data Index: 257528 (C4 (Web Text))

Max Activating Token Index: 717

Text #16

Max Range: 0.2090. Min Range: -0.2090

Max Act: 0.1910. Min Act: -0.0000

Data Index: 1079454 (C4 (Web Text))

Max Activating Token Index: 357

Text #17

Max Range: 0.2090. Min Range: -0.2090

Max Act: 0.2040. Min Act: -0.0000

Data Index: 414122 (C4 (Web Text))

Max Activating Token Index: 143

Text #18

Max Range: 0.2090. Min Range: -0.2090

Max Act: 0.1847. Min Act: -0.0000

Data Index: 705284 (C4 (Web Text))

Max Activating Token Index: 727

Text #19

Max Range: 0.2090. Min Range: -0.2090

Max Act: 0.1699. Min Act: -0.0000

Data Index: 683310 (C4 (Web Text))

Max Activating Token Index: 920