Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4163 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.3059. Min Range: -2.3059

Max Act: 2.3059. Min Act: -0.0000

Data Index: 1144179 (C4 (Web Text))

Max Activating Token Index: 342

Text #1

Max Range: 2.3059. Min Range: -2.3059

Max Act: 1.9053. Min Act: -0.0000

Data Index: 1348712 (C4 (Web Text))

Max Activating Token Index: 270

Text #2

Max Range: 2.3059. Min Range: -2.3059

Max Act: 1.6892. Min Act: -0.0000

Data Index: 1316575 (C4 (Web Text))

Max Activating Token Index: 959

Text #3

Max Range: 2.3059. Min Range: -2.3059

Max Act: 1.7620. Min Act: -0.0000

Data Index: 815279 (C4 (Web Text))

Max Activating Token Index: 678

Text #4

Max Range: 2.3059. Min Range: -2.3059

Max Act: 1.7340. Min Act: -0.0000

Data Index: 1287125 (C4 (Web Text))

Max Activating Token Index: 298

Text #5

Max Range: 2.3059. Min Range: -2.3059

Max Act: 1.4457. Min Act: -0.0000

Data Index: 237099 (C4 (Web Text))

Max Activating Token Index: 277

Text #6

Max Range: 2.3059. Min Range: -2.3059

Max Act: 1.6497. Min Act: -0.0000

Data Index: 380734 (C4 (Web Text))

Max Activating Token Index: 404

Text #7

Max Range: 2.3059. Min Range: -2.3059

Max Act: 1.6224. Min Act: -0.0000

Data Index: 603487 (C4 (Web Text))

Max Activating Token Index: 851

Text #8

Max Range: 2.3059. Min Range: -2.3059

Max Act: 1.5720. Min Act: -0.0000

Data Index: 651088 (C4 (Web Text))

Max Activating Token Index: 129

Text #9

Max Range: 2.3059. Min Range: -2.3059

Max Act: 1.5952. Min Act: -0.0000

Data Index: 482534 (C4 (Web Text))

Max Activating Token Index: 852

Text #10

Max Range: 2.3059. Min Range: -2.3059

Max Act: 1.4972. Min Act: -0.0000

Data Index: 981275 (C4 (Web Text))

Max Activating Token Index: 762

Text #11

Max Range: 2.3059. Min Range: -2.3059

Max Act: 1.5332. Min Act: -0.0000

Data Index: 1086670 (C4 (Web Text))

Max Activating Token Index: 816

Text #12

Max Range: 2.3059. Min Range: -2.3059

Max Act: 1.6503. Min Act: -0.0000

Data Index: 855461 (C4 (Web Text))

Max Activating Token Index: 791

Text #13

Max Range: 2.3059. Min Range: -2.3059

Max Act: 1.5054. Min Act: -0.0000

Data Index: 639367 (C4 (Web Text))

Max Activating Token Index: 720

Text #14

Max Range: 2.3059. Min Range: -2.3059

Max Act: 1.6476. Min Act: -0.0000

Data Index: 637765 (C4 (Web Text))

Max Activating Token Index: 305

Text #15

Max Range: 2.3059. Min Range: -2.3059

Max Act: 1.5433. Min Act: -0.0000

Data Index: 83635 (C4 (Web Text))

Max Activating Token Index: 264

Text #16

Max Range: 2.3059. Min Range: -2.3059

Max Act: 1.5305. Min Act: -0.0000

Data Index: 337993 (C4 (Web Text))

Max Activating Token Index: 688

Text #17

Max Range: 2.3059. Min Range: -2.3059

Max Act: 1.5272. Min Act: -0.0000

Data Index: 1055922 (C4 (Web Text))

Max Activating Token Index: 177

Text #18

Max Range: 2.3059. Min Range: -2.3059

Max Act: 1.5016. Min Act: -0.0000

Data Index: 628820 (C4 (Web Text))

Max Activating Token Index: 825

Text #19

Max Range: 2.3059. Min Range: -2.3059

Max Act: 1.5410. Min Act: -0.0000

Data Index: 1240223 (C4 (Web Text))

Max Activating Token Index: 270