Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2504 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.2235. Min Range: -1.2235

Max Act: 1.2235. Min Act: -0.0000

Data Index: 98877 (C4 (Web Text))

Max Activating Token Index: 643

Text #1

Max Range: 1.2235. Min Range: -1.2235

Max Act: 1.1597. Min Act: -0.0000

Data Index: 323814 (C4 (Web Text))

Max Activating Token Index: 275

Text #2

Max Range: 1.2235. Min Range: -1.2235

Max Act: 1.2010. Min Act: -0.0000

Data Index: 225927 (C4 (Web Text))

Max Activating Token Index: 228

Text #3

Max Range: 1.2235. Min Range: -1.2235

Max Act: 1.0800. Min Act: -0.0000

Data Index: 539829 (C4 (Web Text))

Max Activating Token Index: 723

Text #4

Max Range: 1.2235. Min Range: -1.2235

Max Act: 0.9747. Min Act: -0.0000

Data Index: 95149 (C4 (Web Text))

Max Activating Token Index: 108

Text #5

Max Range: 1.2235. Min Range: -1.2235

Max Act: 1.0203. Min Act: -0.0000

Data Index: 481706 (C4 (Web Text))

Max Activating Token Index: 296

Text #6

Max Range: 1.2235. Min Range: -1.2235

Max Act: 0.9798. Min Act: -0.0000

Data Index: 668412 (C4 (Web Text))

Max Activating Token Index: 378

Text #7

Max Range: 1.2235. Min Range: -1.2235

Max Act: 0.9806. Min Act: -0.0000

Data Index: 855555 (C4 (Web Text))

Max Activating Token Index: 199

Text #8

Max Range: 1.2235. Min Range: -1.2235

Max Act: 1.0119. Min Act: -0.0000

Data Index: 1213275 (C4 (Web Text))

Max Activating Token Index: 852

Text #9

Max Range: 1.2235. Min Range: -1.2235

Max Act: 0.9669. Min Act: -0.0000

Data Index: 506628 (C4 (Web Text))

Max Activating Token Index: 298

Text #10

Max Range: 1.2235. Min Range: -1.2235

Max Act: 0.9450. Min Act: -0.0000

Data Index: 1074675 (C4 (Web Text))

Max Activating Token Index: 184

Text #11

Max Range: 1.2235. Min Range: -1.2235

Max Act: 0.9551. Min Act: -0.0000

Data Index: 504618 (C4 (Web Text))

Max Activating Token Index: 875

Text #12

Max Range: 1.2235. Min Range: -1.2235

Max Act: 0.9396. Min Act: -0.0000

Data Index: 464302 (C4 (Web Text))

Max Activating Token Index: 624

Text #13

Max Range: 1.2235. Min Range: -1.2235

Max Act: 0.9552. Min Act: -0.0000

Data Index: 764245 (C4 (Web Text))

Max Activating Token Index: 36

Text #14

Max Range: 1.2235. Min Range: -1.2235

Max Act: 0.9920. Min Act: -0.0000

Data Index: 1112208 (C4 (Web Text))

Max Activating Token Index: 180

Text #15

Max Range: 1.2235. Min Range: -1.2235

Max Act: 0.9496. Min Act: -0.0000

Data Index: 639481 (C4 (Web Text))

Max Activating Token Index: 11

Text #16

Max Range: 1.2235. Min Range: -1.2235

Max Act: 0.9448. Min Act: -0.0000

Data Index: 1304379 (C4 (Web Text))

Max Activating Token Index: 736

Text #17

Max Range: 1.2235. Min Range: -1.2235

Max Act: 0.9138. Min Act: -0.0000

Data Index: 450103 (C4 (Web Text))

Max Activating Token Index: 863

Text #18

Max Range: 1.2235. Min Range: -1.2235

Max Act: 0.9108. Min Act: -0.0000

Data Index: 1304374 (C4 (Web Text))

Max Activating Token Index: 805

Text #19

Max Range: 1.2235. Min Range: -1.2235

Max Act: 0.9183. Min Act: -0.0000

Data Index: 968965 (C4 (Web Text))

Max Activating Token Index: 897