Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 2430 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 4.4020. Min Range: -4.4020

Max Act: 4.4020. Min Act: -0.0000

Data Index: 314553 (The Pile)

Max Activating Token Index: 338

Text #1

Max Range: 4.4020. Min Range: -4.4020

Max Act: 4.0978. Min Act: -0.0000

Data Index: 873579 (The Pile)

Max Activating Token Index: 135

Text #2

Max Range: 4.4020. Min Range: -4.4020

Max Act: 3.2292. Min Act: -0.0000

Data Index: 521822 (The Pile)

Max Activating Token Index: 517

Text #3

Max Range: 4.4020. Min Range: -4.4020

Max Act: 3.4236. Min Act: -0.0000

Data Index: 1733766 (The Pile)

Max Activating Token Index: 241

Text #4

Max Range: 4.4020. Min Range: -4.4020

Max Act: 3.1314. Min Act: -0.0000

Data Index: 449899 (The Pile)

Max Activating Token Index: 357

Text #5

Max Range: 4.4020. Min Range: -4.4020

Max Act: 3.3846. Min Act: -0.0000

Data Index: 1618648 (The Pile)

Max Activating Token Index: 206

Text #6

Max Range: 4.4020. Min Range: -4.4020

Max Act: 3.5139. Min Act: -0.0000

Data Index: 677634 (The Pile)

Max Activating Token Index: 341

Text #7

Max Range: 4.4020. Min Range: -4.4020

Max Act: 3.1963. Min Act: -0.0000

Data Index: 1382381 (The Pile)

Max Activating Token Index: 835

Text #8

Max Range: 4.4020. Min Range: -4.4020

Max Act: 3.2233. Min Act: -0.0000

Data Index: 1440623 (The Pile)

Max Activating Token Index: 46

Text #9

Max Range: 4.4020. Min Range: -4.4020

Max Act: 3.1895. Min Act: -0.0000

Data Index: 428617 (The Pile)

Max Activating Token Index: 106

Text #10

Max Range: 4.4020. Min Range: -4.4020

Max Act: 3.3683. Min Act: -0.0000

Data Index: 1495617 (The Pile)

Max Activating Token Index: 572

Text #11

Max Range: 4.4020. Min Range: -4.4020

Max Act: 3.0587. Min Act: -0.0000

Data Index: 1660566 (The Pile)

Max Activating Token Index: 502

Text #12

Max Range: 4.4020. Min Range: -4.4020

Max Act: 2.7173. Min Act: -0.0000

Data Index: 568752 (The Pile)

Max Activating Token Index: 585

Text #13

Max Range: 4.4020. Min Range: -4.4020

Max Act: 2.8691. Min Act: -0.0000

Data Index: 192037 (The Pile)

Max Activating Token Index: 744

Text #14

Max Range: 4.4020. Min Range: -4.4020

Max Act: 3.3341. Min Act: -0.0000

Data Index: 568816 (The Pile)

Max Activating Token Index: 411

Text #15

Max Range: 4.4020. Min Range: -4.4020

Max Act: 3.0238. Min Act: -0.0000

Data Index: 1798603 (The Pile)

Max Activating Token Index: 297

Text #16

Max Range: 4.4020. Min Range: -4.4020

Max Act: 3.1261. Min Act: -0.0000

Data Index: 599212 (The Pile)

Max Activating Token Index: 819

Text #17

Max Range: 4.4020. Min Range: -4.4020

Max Act: 3.1278. Min Act: -0.0000

Data Index: 1914211 (The Pile)

Max Activating Token Index: 465

Text #18

Max Range: 4.4020. Min Range: -4.4020

Max Act: 3.5746. Min Act: -0.0000

Data Index: 150779 (The Pile)

Max Activating Token Index: 362

Text #19

Max Range: 4.4020. Min Range: -4.4020

Max Act: 2.9598. Min Act: -0.0000

Data Index: 1770017 (The Pile)

Max Activating Token Index: 505